Hacker News new | past | comments | ask | show | jobs | submit login
Zimg – A lightweight and high-performance image storage and processing system (github.com/buaazp)
196 points by lonre on Sept 9, 2014 | hide | past | favorite | 21 comments



This looks great!

Just a suggestion, but I think you could boost the performance of simple JPEG transformations (crop, rotate by multiples of 90 degrees, transpose/transverse) by using the transformation facilities of libjpeg or libturbojpeg (which has a far nicer API). For scaling there is also epeg which works directly with the DCT coefficients.

I created a Python library that uses these APIs and it beats all ImageMagick-based solutions by an order of magnitude: https://github.com/jbaiter/jpegtran-cffi


Isn't imagemagick a problem for a high perfomance site. I did never find a solution for the GB sized temp files which are created by imagemagick sometimes. I had better results with http://www.vips.ecs.soton.ac.uk .


I tried it. Compiled fine. The example conf file has some chinese comments in it. Most of the basic configuration is pretty obvious.

I started the server and it serves an index.html with an upload form which is nice. The upload response is an HTML page, too which states in the body where the uploaded file is accessible.

There I got curious… how would one use zimg uploads from app? I think the common use case would be: User upload images to a web app or an API app which then POSTs the image to zimg and puts the image location into a DB.

I tried POSTing images using curl having the vague hope the response would be something more machine readable than an HTML body. At first that seemed to be true because I used the wrong curl parameters and zimg couldn't find the image within the POST body and responded {"ret":false,"error":{"code":1,"message":"File type not support."}}. But once I got the curl parameters right (curl -XPOST --form userfile=@9393.jpg -i http://localhost:4869/upload.json) zimg again responded with HTML. Adding an accept header and/or a ".json" postfix to the URL didn't make a difference. Am I missing something?

Is there a possibility to restrict upload to local network only but allowing public download?


Of course. The documents of zimg is written in Chinese so you don't know how to use.

1. Using curl and multiform way can post file to zimg and will return a html result:

curl -F "blob=@testup.jpeg;type=image/jpeg" "http://127.0.0.1:4869/upload"

If you don't like upload files via multiform way, you can use raw post way. Use this command to upload image via raw post:

curl -H "Content-Type:jpeg" --data-binary @testup.jpeg "http://127.0.0.1:4869/upload"

You need tell zimg a header "Content-Type", the value is the type of image. This way will get a json result and the error message is in the table below:

{"ret":false,"error":{"code":0,"message":"Internal error."}} {"ret":false,"error":{"code":1,"message":"File type not support."}} {"ret":false,"error":{"code":2,"message":"Request method error."}} {"ret":false,"error":{"code":3,"message":"Access error."}} {"ret":false,"error":{"code":4,"message":"Request body parse error."}} {"ret":false,"error":{"code":5,"message":"Content-Length error."}} {"ret":false,"error":{"code":6,"message":"Content-Type error."}} {"ret":false,"error":{"code":7,"message":"File too large."}}

2. Upload and download access control. Just modify conf file like this:

upload_rule = 'allow 127.0.0.1;allow xx.xx.xxx.xxx/xx' download_rule = 'allow all'

The conf rule is like nginx.

3. Sorry for missing English documents.


Nice! Thanks. A perfect fit for my current project.


There are some demonstrations on this page: http://zimg.buaa.us/documents/

It feels very fast—has anyone worked with this who can speak to its effectiveness in a production environment?


It indeed seems super fast; when you toy with the rotation parameter in the first example the result is instantaneous.

It'd be interesting to know who uses this.


Are the demo pages accessible on your side? I can't at the moment. I sent a notice to the author.


Nevermind, it is already back online.



It's a nice coincidence that I was trying to find something like this, I was originally going to use nginx+lua to do this job. However, zimg doesn't let us use a custom short url, so to make a website like imgur it would require nginx frontend, which defeats the purpose of using it over nginx+lua in my situation - does zimg's ability to use LUA allow us to get a key from redis which gets the SHA1 of the original image to then use the image storage?

Also, does Zimg provide a way to expire images if they haven't been used in 1 year - and if there isn't enough storage for anymore images, it deletes the last accessed image to create room (I know this can be done in nginx, maybe nginx is still the solution I need)?

Great project!


Interesting requirements. You can send me an email with more details, I will try to implement them. zp@buaa.us


Why does it depend on hiredis (client to Redis) if it is not using redis (at least according to architectural diagram)? Is it just for the twemproxy?


Users can choose which storage to use in config file. Beansdb for memcached protocol and SSDB for redis protocol. SSDB is an alternative to Redis. Or you can use any other storage backend which is using these two protocols.


I believe it uses redis as one of the standard storage methods, but can interchangably use memcached, ssdb and beanydb. It's a bit hard to be sure from google translate documentation though.


SSDB compatible with redis protocol


I think this is a great project but I think that the true problem with image processing these days is infrastructure. Image processing is a rather compute heavy task when you start seeing an influx of traffic. I'll openly announce that I'm one of the founders at http://6px.io, a hosted solution for image processing.


This looks seriously awesome. Though, instead of investing all the time, effort, and resources it would take to use this you could use something like 6px.io and keep your focus on your core product.


Have you considered supporting the International Image Interoperability Framework (http://iiif.io/) API?


Looks very useful, but I wish it used OpenCV instead of ImageMagick. ImageMagick has major memory leak issues.


This is pretty cool. Companies like imgur could benefit from this.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: