We've retooled the 3D Building Pipeline

This is a long post, so grab your favorite beverage and settle in.

Over the last 6 months we've been hard at work "retooling" the 3D building pipeline. We use the phrase "3D pipeline" to describe the process a model goes through when it's been published to the Google 3D Warehouse (assuming it was marked "Google Earth-ready").

The now outdated process for reviewing and publishing your 3D building models to Google Earth's 3D Buildings layer involved a number of manual steps and was inconsistent in terms of release dates. Further, and more importantly, far too many models were getting "lost" in the pipeline and remained in an idle state for long periods of time. We're keenly aware of how frustrating it is to publish a model and wait for Google to review and publish it. So, late last year, we set off on a mission to resolve these issues.

While this may appear simple on the surface, I can assure you that it's not. You might be surprised at the number of steps a model goes through from your publishing of it, to the model appearing in Google Earth. We like to be as transparent about the process as we can, so here is a high-level overview of those steps for your inquiring minds:

  • KMZ creation - The very first thing that we do when you publish a model to the 3D Warehouse is create a KMZ file. This allows you and others to download/view your model in Google Earth.
  • Model review - Next, a team of people review the model to ensure that it adheres to our acceptance criteria (more about that later).
  • Duplicate check - In this step, we identify where duplicate and/or overlapping models exist. There's more of these than you might imagine, like 163 Eiffel Towers for example. This is also where we need to make decisions about which model is the best, and why some folks see a "better model exists" appear as a judgment on their model detail page . You'll be happy to know that when a user-provided model exists that is equal to or better than an auto-generated model, we will always choose the user model.
  • Test build - After checking for duplicates, we "ingest" all of the data that has passed through the previous step to a test server where it receives another review. A small number of models will actually fail in this step for a few different reasons - it may be too complex for example, or have bad textures. While the evaluations prior to this point in the review process have been visual, this step checks for issues that might effect display, performance, or both. It's also where we make sure that cities are where they belong, that no space elevators shoot into space, and no Godzillas appear in Tokyo.
  • Build and push - Finally, we push the models into our data centers so that they appear in your Google Earth client. Our data centers are spread out around the world so that, wherever you are, you can have a good experience in Google Earth.
What does all this mean to you? The new, automated, 3D pipeline will allow us to review and publish your models faster, and with far greater predictability. Specifically, the new system will enable us publish data on a *weekly* basis and, in the near future, even faster. So, with the exception of the one week per month where we need to conduct maintenance, we plan to publish models each and every week going forward.

Lost Models
Prior to this effort it was far too easy to find a model in the 3D Warehouse that appeared to be in a perpetual state of "awaiting review." Some models have been in this state for months or even years (ouch). So, in addition to retooling the process, we also scrubbed the 3D Warehouse for each and every geo-located model and sent it through our review process again. We scrubbed, and scrubbed some more, to make sure that every model that deserved to be in Google Earth was in Google Earth. Some of you noticed that a large number of models appeared in the layer in May because of this change. In fact, since we began this effort, we've *doubled* the number of user-generated models that appear in Google Earth today.

At this point we've completed our scrub and we're confident that every geo-located model has been reviewed. OK, full-disclosure.... there are a number of models that, for a variety of reasons, aren't added to the 3D Buildings layer *after* they have passed through our initial review (see "Test build" above). They look great visually, but have some other issue that may be preventing them from being successfully ingested into the layer. But, if you happen to own one of these models, fear not, we're working to identify what may be causing this. At a minimum, we'll want to update the feedback mechanism to make you aware that there's an issue with the model that's preventing it from appearing in Google Earth.

Acceptance Criteria
While some models have been overlooked, many others have been judged more rigorously than we'd like, so we modified our acceptance criteria to be a bit more lenient than it had been previously. We also developed and documented training material to ensure that we were applying a consistent approach in reviewing your models. It's still imperfect but I'm confident that it's far less subjective than it may have been before. As noted previously, we then ran all models that had been rejected through the pipeline again.

"So how has the acceptance criteria changed?" you ask. Good question. We plan to introduce some sample collections very soon that will serve as examples for what we deem acceptable, and highlight how to improve models that may need a little more work. It's not an easy job and it's one that we take very seriously. We're constantly balancing our desire to include as many models as possible, with an effort to establish a level of quality where, when combined, all of the 3D building data is visually harmonious. Geo-modeling is quite different than traditional SketchUp modeling, and we're seeing an increasing number of really fantastic phototextured models being published. Geo-modeling is definitely catching on!

In closing, we have several other features that we'll share with you soon, but suffice it to say that we've heard your pleas on the 3D Warehouse Group for faster turnaround and greater transparency in our process; we hope that you'll be pleased with the steps we've taken thus far.

Permalink | Links to this post |
The comments you read here belong only to the person who posted them. We do, however, reserve the right to remove off-topic comments.

21 comments :

Lorenz Eisermann said...

Thank you very much for clarifying this, Bruce. While geo-modelers always have to guess what will become of their models after submission it's more of an educated guess now. A consistent pipeline leads to results that can be reproduced any time. I think, if the model from the automatically generated KMZ file looks somewhat close to its appearance on the 3d layer of Google Earth most of us will be happy. In-depth info on how your machinery compresses image files, simplifies geometry and handles meta data could help a lot to understand how we can build better models.

Regards,
Lorenz

pmolsen said...

The new "Tell us why we are wrong" link does not work. The drop down boxes disappear off the right hand side of the page and are unreadable. When I click Submit it pops up a message in the bottom left saying Error on page.

Mike Lucey said...

This is great news Bruce and I can fully appreciate the hard work that you guys are doing behind the scenes to make things work for us.

I think this new 'fast flow' will definitely help modelers wishing to try and commercialise their work, well done to all!

Mike Lucey
www.SketchUcation.com

Auquicu said...

Thank you for the improvements.
I would like to know the criteria applied to the use of translucent materials (paint tool) in models for the 3D layer in GE. 2 of my models had translucent materials being replaced by solid materials when included in the 3D layer, so I supponse translucent materials are not desired, but I would like to have I clear so it would be nice to have a mention about this in the new criteria.
The coffee cup attraction in Disney Paris has it's roof made of a translucent material. Does a special criteria applies to sponsored models?

Thanks, Auquicu.

fmblog said...

hi
i think that this approach is wrong, the process should focus more on the indvidual user then on data.
aswell as it should differentiate between users who create for fun and those who are willing to release some serious quality work.

dont get me wrong i really appretiate all efforts,
but creating a good looking model is far more
complex then some suggest.
(eg texture/speed/lod/roof/height)

Contrarian said...

I'm sure this new system will work well once tweaked. The last few layer updates, however, have included some models which do not come close to meeting Google's stated acceptance criteria --- some which are completely untextured, for example. See the "Whoops" thread in the 3D Warehouse Sketchup subforum for a fairly long list with links.

Google said...

Hi Folks,

Thanks for your comments and questions. I consciously avoided diving into to too much technical detail in the post - it was fairly verbose as it was.

For the technically curious, there are several optimization checks that occur in the "test build" step that I outlined in my post. Specifically:

1) Texture LOD optimization. Image textures are downsampled to a maximum of 1 pixel / 2 cm, with a texture size limit of 2048x2048 regardless of dimension of the object to which the texture is applied.

2) Extremely heavily tiled textures (texture coordinates > 10000) will fail processing and will receive a "model too complex" judgement.

3) Less extremely tiled textures, due to 1, will look like their average pixel / meter dimension is very very large (e.g. 1 pixel / mm or worse). The texture can then get downsampled to the point where it no longer renders as an image, but as a uniformly colored blur. This is normally a modeling error (pasting a texture onto a surface unintentionally).

4) Geometric compression. Parallel surfaces less than a few inches apart will be collapsed into one surface. This can cause geometric anomalies (when the wrong two surfaces get squished together) and Z-fighting.

5) You should not attempt to place finely-detailed geometry into Google Earth.

Note that none of this optimization is done "by hand" by Google people. It's all done by machine.

Finally, please keep in-mind that these numbers (e.g. max resolution, geometry quantization) are variable and may change in the future.

Happy Sketching.

Bruce

Google said...
This comment has been removed by the author.
Google said...

Hello again. It was pointed out that I did not address the transparency question from "Auquicu". Here's the scoop:

SketchUp lets you define a texture to be "partially transparent" (opacity < 100%). People use this for windows, texture overlays, and other things. It shows up in the COLLADA file as a value between 0 and 1 for the [transparency] element. The live Google Earth rendering engine ignores this field entirely, so these textures end up looking opaque. The only way to get a live Google Earth model to have transparency is to use a transparent .png texture with whatever alpha you wanted baked into the texture. This is what the Disney models (and all other models currently visible with a transparent or semitransparent material in the 3D buildings layer) use.

Hope that helps. Happy Sketching.

Bruce (with assistance from Allison)

pmolsen said...

What do you me re use of .png texture for translucent textures?

I just used the tools provided in Sketchup for creating a partially transparent colour in the top fin of my Q1 model at http://sketchup.google.com/3dwarehouse/details?mid=3b86d7995aac79079cf69f6066d75cb8.

Are you saying that what Sketchup itself provides is not recognised by GE???

How exactly should it be created please? (Can Paint.NET do it?)

pmolsen said...

Here's another question. I have two very extensive models that I was told by Sketchupguide Tommy back on 25 Nov 2008 were going to be included in the 3D layer within 4 weeks. It never happened, then the entire terrain in the areas changed and they are now totally wrong. Floating/sunken etc.

It will take at least a month to fix each of them. Before I do that is there any way with the new process that I can get them reviewed, ignoring the floating/sunken problems, so that I know that if I spend the time to fix them they will then be acceptable and will not be rejected for some other reason such as too complex? I tried to use "Tell us why we are wrong" but it doesn't work.

(They are http://sketchup.google.com/3dwarehouse/details?mid=56273db983ef92c710ddc3e7148114b6
and
http://sketchup.google.com/3dwarehouse/details?mid=14af8e887bdd4f809cf69f6066d75cb8)

pmolsen said...

Forgot to mention, both of the models show as being in the "Best of 3D Warehouse Nominees" collection.

Auquicu said...

I have just uploaded a new version of the model I made with transparencies, now with PNG instead of JPG. Can you check it can be uploaded to the 3d layer in GE, replacing the older model???
The new one is here:
http://sketchup.google.com/3dwarehouse/details?mid=92965b7524477f0b11166d4226b8ac2d

Mapper99 said...

This sounds like an efficient process. Is there any chance you will start using Laser Data and Imagery from Google Street View to create these models?

Mapper
Street View Gallery

Trasancos said...

It's nice to read that you're improving the acceptance process. Those are great news!, thanks!

I have some doubts. When I update or improve some of my own models (accepted in 3D layer yet), the old version still remains in Google Earth, even when the new version appears in the 3D warehouse.

There are some way to report updates in models of the 3D layer?, or we only have to wait for a new review?

Regards and thanks for your attention.

You're making a great job!

Google said...

Hi Folks,

Thanks for your comments. If you have questions about my posts, please post them on the Google Group which is better suited for discussion.

Thanks!

Bruce

Allison Floyd said...

As Bruce said, this will be easier if you take your questions over to the groups, but while I'm here:

pmolsen: The two models that you're referring to are really large. You'll need to split them into several smaller models for them to get through.

Auquicu: Your model now has correct transparency.

Trasancos: The re-review process is not instantaneous, although it should be much faster now. Your edits will get reviewed automatically, you don't have to do anything special.

-- Allison

Auquicu said...

Thank you Allison. I'm afraid that my model still doesn´t have transparencies. I kept the old model in the 3d Warehouse and uploaded a new model with png in http://sketchup.google.com/3dwarehouse/details?mid=92965b7524477f0b11166d4226b8ac2d&prevstart=0
If the update was made at the old model location, no changes will be made as it has the JPG non working transparencies. Please check, and thank you for the good work and improvements.

pmolsen said...

Allison, I have recently posted 5 posts on the Groups discussion but nobody from Google responds.
http://groups.google.com/group/3dwh/browse_thread/thread/fe1cf2d58707c119?hl=en

http://groups.google.com/group/SketchUp3d/browse_thread/thread/6bdba4116aca46ca?hl=en

http://groups.google.com/group/3dwh/browse_thread/thread/3fe5b4d325b2f5f2?hl=en

http://groups.google.com/group/Sketchup-Pro/browse_thread/thread/a1ce9144c1f5f277?hl=en

Plus one about "Tell us why we're wrong" not working.

Chandresh Kumar Bind said...

Hi

this is my first blog so please i thinks the Google is given good response to this

seoonlineanamikaservice said...

thanks