Paper: 2611.26586 · ProtoDiffusion: Token-per-Layer Generative Modeling of Multi-Layer Images
Authors: Adrià Ciurana Lanau, Harleen Hanspal, Lusine Abrahamyan, Emilio Tylson Baixauli, Yu Cao, Danai Triantafyllidou, Stefano Zorzi, Gustavo Zanoni Felipe, Peter Griger, Stefan Sietzen, Petru-Daniel Tudosiu
Category: cs.AI
Reading scope: Original arXiv v1 HTML, including the method, results tables, limitations, and relevant appendices. Figure captions were read; figure images were not visually inspected. Results were not independently reproduced.

The Problem

Imagine a travel poster with a big title laid over a mountain skyline. It looks finished until someone asks you to move the title. A flat export gives you pixels, so first you have to isolate the letters, then fill in the sky they hid. The exact pixels behind the title were never visible. An editable poster gives you a cleaner starting point: the text and the sky are separate layers.

Some models can already return layers, but Qwen-Image-Layered (QIL) gives each layer its own full canvas representation and asks for a layer count. A region-based system such as MRT can use smaller grids, but needs a layout. The ProtoDiffusion team asks whether the model can choose layers as it draws, without giving every extra item its own copy of the canvas (§§1–2).

The Explanation

One canvas, several ways to read it

Start with a poster whose layers are already known. ProtoVAE, the paper’s compressor, reads them together and saves two things: a shared grid that carries spatial information about the whole design, and a compact vector for each layer, background included. A vector is just a short list of numbers. The title’s vector is produced in the context of the other layers, so it can help the decoder pick the title out of the shared grid. The sky’s vector tells the same decoder to produce the background. The grid is learned; it is not the flattened poster. The training target includes complete layers, including areas that other layers cover (§3.1).

Now start without the editable original. Given either the flat poster or a text brief, ProtoDiffusion builds the shared grid and the candidate vectors together. As it turns noise into a design, the two parts exchange information: what belongs in the title and where the title goes can develop together. Optional boxes give the model locations to work with. Without boxes, the model predicts placement; a score on each candidate decides which slots to keep. The default pool has 50 slots, and unused ones are trained to look empty. The algorithm still processes those 50 slots even if the brief calls for a simpler design (§3.2).

What the poster tests reveal

Table 2 asks how well models recover layers from flat Crello designs. With no adjacent layers allowed to merge before scoring, ProtoDiffusion’s color error was 0.040 with boxes and 0.057 without, compared with 0.059 for the fine-tuned QIL baseline; lower is better. Its transparency overlap with the original layers, where higher is better, was 0.850 with boxes and 0.681 without, compared with 0.871 for QIL. The box-free version can get the foreground color slightly closer while agreeing much less on where the layer starts and ends. For a title you plan to move, the second measure matters.

Table 3 starts from a brief instead. On the authors’ H200 test, ProtoDiffusion took a median 11.58 seconds to generate layers. The pipeline that drew a flat image with FLUX.2 [klein] and then decomposed it with QIL took 220.86 seconds, about 19 times as long. FLUX alone took 9.63 seconds and produced a flat image. These are end-to-end, one-design timings on 50 briefs, excluding model loading and evaluation. On a separate set of prompts, the recombined ProtoDiffusion images also had a lower FID, a measure of how close a collection of images is to a reference collection: 66.41 versus 86.84 for FLUX alone. That score cannot tell you whether the title came out on a layer of its own.

Core Concepts

A layer carries its own transparency. An RGBA image stores color and a transparency value for each pixel. The title layer should be transparent between the letters. Otherwise, dragging the title drags patches of sky along with it.

The shared grid pays for space once. Before packing for the image model, storing one full grid per layer multiplies the grid size by the number of layers. ProtoVAE adds one vector per layer to a single grid. The denoiser processes the entire candidate pool, while the final decoder still has to turn every retained candidate into a separate layer (§3.3).

The slot threshold controls the output count. Raising the threshold can discard a real title; lowering it can keep empty or redundant layers. A slot score helps decide what to decode, but it cannot tell you whether every element a designer cares about survived.

The Insight

When the result has to stay editable, the units you will edit should survive generation. Two posters can look the same on screen and give their designers very different work the moment a title moves.

A Critical Reading

The paper provides separate layer and composite measures, which makes the method’s tradeoff visible. The box-free model does not reliably reproduce the source design’s layer count, either. In the post-hoc calibration study, the threshold chosen on one split matched the exact count in 12.8% of examples on the other; 77.99% had fewer retained layers than the original. A lower count alone does not establish that an element disappeared: equivalent compositions can organize components differently. An element-level check would show which differences interfere with actual editing (Appendix B.1).

The speed comparison tells us about those pipelines and settings. The layer-count scaling test uses different backbones and inputs across systems, so it cannot isolate how much of the gain comes from the shared grid (Fig. 6; Appendix B.2). The authors also flag very dense overlaps, thin elements, and transparency leakage as open cases (§5). And for a flattened input, the true hidden sky is underdetermined; what the model reveals is a learned reconstruction.

What You Can Use

When you evaluate a generator for design work, give it a poster with overlapping objects. Move the title, turn off the foreground, and swap in a new background. Check for clipped letters, holes behind the title, and stray pixels on the type layer. Try the same edits with and without boxes if your workflow supports both. A single before-and-after screenshot of the composite can’t answer those questions.

论文:2611.26586 · ProtoDiffusion: Token-per-Layer Generative Modeling of Multi-Layer Images
作者:Adrià Ciurana Lanau、Harleen Hanspal、Lusine Abrahamyan、Emilio Tylson Baixauli、Yu Cao、Danai Triantafyllidou、Stefano Zorzi、Gustavo Zanoni Felipe、Peter Griger、Stefan Sietzen、Petru-Daniel Tudosiu
分类:cs.AI
阅读范围:阅读 arXiv v1 原始 HTML 中的方法、结果表、局限与相关附录。读了图注,未查看图像本身;未独立复现实验。

问题

一张旅游海报,标题压在山脊上,成图挺好看。有人说:「把标题往右挪一点。」如果手里只有一张导出的平面图,你先得把字抠出来,再补上字底下那片天。可那片天原来长什么样,成图从来没给你看过。手里有图层就省事:字是字,天是天,挪一个不用重画另一个。

已有模型会拆图层。Qwen-Image-Layered(QIL)给每一层各存一张完整的空间网格,还要你先报层数;MRT 可以只存局部网格,但得先有版面位置。ProtoDiffusion 团队于是追问:能不能让模型一边画,一边定下该拆几层,又不必每添一件东西就添一张画布?见第 1—2 节。

翻译

一张底稿,各层有各层的读法

先拿一张有原始图层的海报来训练。压缩器 ProtoVAE 把各层放在一起读,只存两样东西:一张共享网格,记下整个版面的空间信息;每层再配一小串数字,背景也有一串。标题的数字是看过其他层之后算出来的,解码器拿它读共享网格,就能还原标题;换成背景那串,还是同一个解码器,读出来的变成了背景。这张网格不是导出的平面海报。训练时,作者拿完整图层给模型对照,连被其他层挡住的部分也算在内,见第 3.1 节。

手里没有原图层时怎么办?给 ProtoDiffusion 一张平面海报,或者一段文字需求,它就从噪声里一起推敲那张网格和每层的数字。标题是什么、应该放在哪,两头可以互相修正。有方框,模型就用方框作位置提示;没有,也能自己猜位置,再按分数留下有内容的图层槽位。默认备 50 个槽位,多余的在训练时学着留空。海报即使只用几层,这 50 个槽位仍要一起处理,见第 3.2 节。

颜色还原得准,边缘未必跟得上

表 2 用 Crello 测试集拆平面海报。在计分前不许合并相邻图层时,ProtoDiffusion 有框和无框的颜色误差分别是 0.040、0.057,微调后的 QIL 是 0.059,越低越好。再量预测图层与原图层的透明度重合,越高越好:有框是 0.850,无框是 0.681,QIL 是 0.871。无框版能把前景颜色还原得稍准一点,却更容易弄错图层边界。平放看不出的问题,挪标题时可能就露了出来。

表 3 从文字需求起步。在作者用 H200 测的 50 份需求里,ProtoDiffusion 生成一张有图层的设计,中位用时 11.58 秒。换成先用 FLUX.2 [klein] 出平面图、再用 QIL 拆层的两段流程,要 220.86 秒,约是前者的 19 倍。只用 FLUX 画平面图则是 9.63 秒。这里计的是单张图端到端耗时,不含加载和评估。另一组提示词测试里,两种输出都按平面图比较:ProtoDiffusion 的 FID 为 66.41,单用 FLUX 为 86.84。这个分数看一批成图和参考图有多接近,越低越好;它看不出标题是否独占一层。

核心概念

图层自带透明度。 RGBA 图层除了存颜色,还记每个像素有多透明。笔画之间应该透出天空;如果没留干净,拖字就会把一块块天也拖走。

共享网格只存一遍位置。 在给图像模型打包之前,每层各用一张完整网格,层数翻倍,空间格子也跟着翻倍。ProtoVAE 保留一张网格,每多一层只添一串数字。生成时模型仍要处理整组候选槽位;最后留下多少层,就得分别解码多少层,见第 3.3 节。

槽位分数只能决定留不留。 门槛高了,真标题可能被当成空位;低了,又可能留下空层或重复层。一个分数能决定是否解码这层,却回答不了设计师要改的元素有没有都留下。

洞见

要让成品能改,就得在生成时留下将来要改的东西。两张海报摊开一模一样,标题一挪,才知道哪张真的好用。

博导审稿

把成图和图层分开量,这一步做对了,也让无框方案的局限看得更清楚。事后校准槽位门槛时,作者用一半样本选阈值,在另一半上检查,输出层数与原稿完全一致的比例只有 12.8%,而 77.99% 的样本层数比原稿少。层少不一定是漏了元素:看起来相同的设计可以有不同的分层方式。还得逐个检查需要编辑的元素,才知道这种差异会不会妨碍工作,见附录 B.1。

速度数据说明的是这几条流程在测试条件下跑得多快。随层数增长的实验里,各系统用的底座与输入提示不同,不能单凭这张图算出共享网格省了多少时间,见图 6 和附录 B.2。作者也列出了还没解决的问题:图层挤在一起、细线细字,以及边缘漏出不该出现的像素,见第 5 节。至于平面图里没露过面的那片天,模型能补出一个合理版本,却没有办法知道原来恰好是什么样子。

启发

试用设计生成器时,挑一张有遮挡的海报。把标题挪开,把前景关掉,再换一张底图。看看字边有没有缺口、标题底下会不会破个洞、新背景上会不会残留旧背景的碎屑。流程支持方框的话,两种输入都试一次。仅看拼好的整张海报,答不了这些问题。