Google Releases New Compression Algorithm


Recommended Posts

  Quote
The Zopfli Compression Algorithm is a new open sourced general purpose data compression library that got its name from a Swiss bread recipe. It is an implementation of the Deflate compression algorithm that creates a smaller output size compared to previous techniques. The smaller compressed size allows for better space utilization, faster data transmission, and lower web page load latencies. Furthermore, the smaller compressed size has additional benefits in mobile use, such as lower data transfer fees and reduced battery use. The higher data density is achieved by using more exhaustive compression techniques, which make the compression a lot slower, but do not affect the decompression speed. The exhaustive method is based on iterating entropy modeling and a shortest path search algorithm to find a low bit cost path through the graph of all possible deflate representations.

The output generated by Zopfli is typically 3?8% smaller (PDF warning) compared to zlib at maximum compression, and we believe that Zopfli represents the state of the art in Deflate-compatible compression. Zopfli is written in C for portability. It is a compression-only library; existing software can decompress the data. Zopfli is bit-stream compatible with compression used in gzip, Zip, PNG, HTTP requests, and others.

Due to the amount of CPU time required --- 2 to 3 orders of magnitude more than zlib at maximum quality ? Zopfli is best suited for applications where data is compressed once and sent over a network many times, for example, static content for the web. By open sourcing Zopfli, thus allowing webmasters to better optimize the size of frequently accessed static content, we hope to make the Internet a bit faster for all of us.

Source: http://googledevelopers.blogspot.com/2013/02/compress-data-more-densely-with-zopfli.html
  UXGaurav said:
Wonder how it does against LZMA and LZMA2 considering both reasonable speed and compression ratio.
This is about DEFLATE-compatible compressions, so LZMA/LZMA2 don't enter into the picture.

But yes, LZMA/LZMA2 traditionally has superior compression ratio. Too bad that still not used in HTTP transfer-encoding.

I read the PDF. It's decent, but has a very specific use case. That is: One-time compression and multiple decompressions for static data (they do acknowledge this). Compressing files using this algo is epic slow compared to gzip for the same file (talking 5 seconds for gzip compared to 7 minutes for the same file using Zopfli). It'll be very good for conserving bandwidth on, say, file distribution sites though where CPU time is inconsequential compared to bandwidth requirements.

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • only works for 1 microsoft account . im on rp channel and have 2 pcs with the same microsoft account, i pc got it the other one did not
    • Your first comment about one needing to send all their data to Microsoft's "cloud" to continue Windows 10 support was of course ridiculous but I can't disagree here. I am not sure why the OP would wait to switch to Linux if their Mom just uses a browser as she can use the same one with Linux. Why would she care what OS is running? Install /configure Linux Mint or Ubuntu LTS and almost no maintenance really.
    • Seagate Exos 20 TB BarraCuda 16 TB 7200 RPM CMR hard disk drives are limited-time deals by Sayan Sen Currently, there are some great sales on SSDs, in particular, NVMe M.2 ones, like this SK hynix P41 Platinum P41. However, if you are in the market for a hard disk drive (HDD), then Seagate has you covered. That is because two high-capacity drives, the Exos X20 20TB and the BarraCuda 16 TB are currently at great prices (purchase links down below). Since both of them are high capacity models, they are based on CMR or Conventional Magnetic Recording technology which is typically great for robust use cases like on NAS or (media) home servers and similar such scenarios. However, the Exos is enterprise-grade quality so it is definitely what you would want for 24x7 usage, but the BarraCuda can also serve pretty well but will be a better option for backup purposes. The Exos X20 promises power-on hours of 8760 per year, while the BarraCuda claims 2400. The latter comes with double the DRAM cache, though with 512 MB of it, while the former has 256 MB, but it should still be plenty. In terms of data transfer speed, Seagate promises up to 285 MB/s (272 MiB/s) from the Exos X20 and the BarraCuda is rated for up to 190 MB/s. These are the maximum sustained data transfer rates for the two drives. Since the Exos X20 is an enterprise-class product Seagate does disclose its random performance throughput, which is 168 IOPS for reads and 550 IOPS for writes, respectively. While the BarraCuda's randoms are not specified it is expected to be lower and to compensate, it packs double the DRAM cache. Get the HDDs at the links below: Seagate Exos X20 ST20000NM007D 20TB SATA 6.0Gb/s 3.5" Internal Hard Drive: $379.00 + $50 off with promo code SACET2372 => $329.00 (Newegg US) Seagate BarraCuda ST16000DM001 16TB SATA 6.0Gb/s 3.5" Internal Hard Drive Drive: $194.99 (Newegg US) This Amazon deal is US-specific and not available in other regions unless specified. If you don't like it or want to look at more options, check out the Amazon US deals page here. Get Prime (SNAP), Prime Video, Audible Plus or Kindle / Music Unlimited. Free for 30 days. As an Amazon Associate, we earn from qualifying purchases.
    • Why do you think the writing is on the wall? Tesla continues to have the highest selling models on the planet, and they continue to grow on that despite their junk reliability rating. I'm not going to buy one because of the continued build quality issues (and lack of Carplay), but I appreciate that they're the best player in an EV market that is massively lacking in infrastructure and affordability. Besides the EV technology needing to eventually outpace gasoline, self-driving technology is undoubtedly going to be the future. It's not going to happen tomorrow and not even in 10 years, but within our lifetimes, most cars on the road will be self-driving. That's the only way to solve the issue of a million deaths on the road. Waymo is using technology that's simply too expensive, which means it has to charge high fares. Tesla is going to blow past them with their ~$40k car and further yet when they release that weird looking 2-seater. Pretty soon Nvidia is going to be a market player, Amazon will be a player, and possibly the other 20+ startups that are working on a vehicle. I'd like to revisit this in a year and see how the market looks. I'm willing to bet there will be a huge increase of driverless cars on the road, and I doubt that Waymo is going to be the majority of them. Waymo isn't profitable for Google, so it's entirely possible the plug gets pulled as soon as several players leave them in the dust.
    • I have always found it funny how so many here complain everyday about Windows but often are the same people who will attack anyone who even suggests using Linux.
  • Recent Achievements

    • Week One Done
      DrRonSr earned a badge
      Week One Done
    • Week One Done
      Sharon dixon earned a badge
      Week One Done
    • Dedicated
      Parallax Abstraction earned a badge
      Dedicated
    • First Post
      956400 earned a badge
      First Post
    • Week One Done
      davidfegan earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      615
    2. 2
      ATLien_0
      229
    3. 3
      Michael Scrip
      167
    4. 4
      +FloatingFatMan
      157
    5. 5
      Xenon
      142
  • Tell a friend

    Love Neowin? Tell a friend!