Euclideon Dreams
Give me something to believe in: Is #Euclideon Unlimited Detail a hoax?
Every revolutionary idea seems to evoke three stages of reaction. They may be summed up by the phrases:
- Its completely impossible.
- Its possible, but its not worth doing.
- I said it was a good idea all along.
- Arthur C. Clarke
Where to begin?
Recently I made a post on Google+ about the future of graphics technology, citing both the Outerra and the upcoming Euclideon Engine. In some regard, they both represent an interesting crossroads in the industry where one company (Outerra) is using the polygon methodology mixed with procedural systems, while Euclideon is using a variant of a voxel based system with procedural methodologies. What transpired was a really interesting debate between myself and one of the staff from Outerra (Brano Kemen) who falls somewhere between the first and second category of debate as listed above from Arthur C. Clarke.
His insistence that Euclideon was likely just a hoax wasn�t what raised my eyebrow, but instead it was the reasoning he was using. In a typical fashion, he put forth his best arguments against Euclideon from the perspective of a current generation technology programmer.
The obvious stance being � Well, this isn�t a new idea and it�s been tried before, but nobody has managed to pull it off like Euclideon claims they have.
I�m a little perplexed at this statement because it�s really no different than saying:
Thousands of people in the past have tried this and failed, therefore it�s not possible that somebody will come along and succeed.
At best, that�s a bit of faulty logic.
Generally speaking, if you�re coming at this from the perspective of what you already know in the industry, then the Euclideon engine looks like nothing more than witchcraft and magic. Of course, Arthur C. Clarke also notably commented on this phenomenon as well:
Any sufficiently advanced technology is indistinguishable from magic.
� Arthur C. Clarke
Past Tense Future
Ive done a few articles on this blog awhile back concerning the Euclideon engine and what I believed at the time were the general mechanics of the engine which made it so powerful (and unique). For the most part, the more I research into this system and read (or listen) to the details that are made available, the more I am convinced it is not a hoax. If you would like to read the original posts on this subject, feel free to hop over to Quantum Rush and Quantum Rush [Redux] for some perspective.
To be fair, the Outerra engine is marvelous in its own right and has definitely earned my respect for what is being accomplished, but in the grand scheme of things, it�s like praising one technology at the height of advancement but knowing it�s a sort of last hurrah! before the sun sets and a new age approaches. I say the same thing about Crytek as well, and even they believe that technology like Euclideon is not a hoax because they tried to import a point-cloud model of a tiger during their research on the Crytek engine.
So in the grand scheme of things, at least one company thinks it�s possible and they sure as hell are bigger than Outerra. That being said, Crytek also acknowledged that they didn�t manage to crack the secret recipe for point-cloud data approaches and went with the approach they have now in the Crysis 3 engine.
Outerra Engine in alpha stages. Ground is procedural, but models are polygon.
In the beginning, was the Pixel
Simply put, the Euclideon engine is not a typical voxel engine. It may use similar foundations from this technology, but that�s about where the similarities end.
In order to wrap our heads around the claim of Unlimited Detail, we first start with the idea that your screen only has a certain amount of pixel space. Let�s say this is your screen resolution and go with 1900x1200 as a default resolution (which is mine).
Now, the first thing we ask is how many pixels is that on the screen?
It�s really a math problem, simple multiplication:
1900 pixels wide by 1200 pixels height.
This math problem yields us an answer of:
2,280,000 pixels on screen.
We can display photographs on screen at 1900x1200 resolution and that still image looks, well, photorealistic. But in a 3D Engine, photorealistic seems to be a word reserved for rendering farms and a year of processing.
In the case of Euclideon, they realized something that I don�t think the rest of the industry realized when dealing with point-cloud data models. Typically we�re loading the entire model into memory and shuffling it around, and that actually does get computationally expensive, but we�re talking about also moving around all the little digital atoms in the model that you can�t see, which is about 90% of the model at any given time.
Why exactly are we shuffling around 90% of a model we can�t see? That means we�re wasting 90% of our processing power doing stuff nobody will notice.
The obvious question then became:
What can the user actually see?
The answer to this is clearly:
They can only see what�s in front of them at any given moment, or whatever pixels are lit up in their resolution.
Well, this gives us something interesting to think about. If the pixel space 1900x1200 is the maximum amount of pixels on screen at any moment, then the only thing we need to know from there is what color those pixels are.
This is just one half of the equation, though. So we set this thought aside for a moment and ask:
How do we figure out what the user can see out of those 2,280,000 pixels on screen?
The World of Tiny Little Atoms
Setting aside the answer of 2,280,000 pixels on screen, we then look to figure out how we deliver only what will satisfy that prior question of what of the models can be seen in that screen space.
This is the point where another explanation from Bruce Dell in a recent interview comes to mind where he explains that current systems are pulling the entire book (model) and processing, yet what happens when you index the individual words in the books properly and make them searchable?
In a point-cloud model, we�re working with little digital atoms, so the model file itself is comprised of atomistic descriptions. The goal, then is to figure out which of those digital atoms in the model file are visible on screen at any given moment and ignore the rest of the model file.
So when you take a model of 500,000 polygons and convert it to the Euclideon point-cloud data format (which for all intents and purposes likely also indexes those points in the file) you end up with a high resolution point-cloud model where every point is indexed for searching. This is where we get the beginning of the second part of our answer for Euclideon and where the typical voxel engines stop.
The Euclideon engine is said to work a lot like a search engine, where the only search query is:
What pixels on the screen correspond to the point-cloud data in the model files that this user can see?
When we do a camera check in-world to see what is in front of the user, the engine likely returns the models it sees, and then the individual points from those files that are visible to the user within the screen space (again, 2,280,000 pixels).
Knowing this information, we ask the point-cloud data models up front to return only the points within it that correspond to the pixels on screen. Since the point-cloud models are pre-indexed to begin with, and the points (atoms) inside of it are individually indexed inside the file, only the points in the model that match the search query are returned as an answer to the engine, ignoring the other 90% of the file up front.
What we are likely left with then is something like this in the scene:
Clearly this is a good start, but we have quite a lot of empty space in between those pixels that are unanswered from just the points in the model files that are stored. Obviously, we wouldn�t want to store the point-cloud data at absolutely full resolution (because the storage then becomes the issue), so there must be a way to fill in the blanks on this point cloud scene.
Luckily for us, we have an answer.
Algorithmic Interpolation
More or less, the idea of what happens in between the points on the point cloud model are likely left to an algorithmic interpolation sequence. If we have two points and there is space between them, then the algorithm takes the average of the two sides and creates a new point algorithmically generated.
This isn�t a new idea, but when applied to something like voxels it gets interesting. The closer you get to the atoms in a voxel display system the more they enlarge and even start generating algorithmic interpolation to fill in the spaces, creating detail where there wasn�t before.
This is a common technique in console emulators to upscale 2D sprite art to modern HD screens, so Mario as sprite art can look good when made bigger on your high definition computer screen. One of the more popular algorithms for this 2D Sprite scaling is the hqx algorithm, and is available as part of most current console emulators.
In image processing, hqx ("hq" stands for "high quality" and "x" stands for magnification) is one of the pixel art scaling algorithms developed by Maxim Stepin, used in emulators such as Nestopia, bsnes, ZSNES, Snes9x, FCE Ultra and many more. There are 3 hqx filters: hq2x, hq3x, and hq4x, which magnify by factor of 2, 3, and 4 respectively. For other magnification factors, this filter is used with nearest-neighbor scaling.
Procedural Methods
In much the same manner as Outerra makes use of procedural methodologies for their terrain, Euclideon seems to make use of procedural methodologies for a majority of the engine as applied to specialized point-cloud data that is indexed efficiently.
It�s like going to Google in order to search for something. Naturally you don�t expect Google to start at the beginning of the Internet and go through everything in a linear manner looking for what you searched for?
No, instead Google crawls the Internet and indexes the content which is why when you search for something it skips 90% of the Internet and brings back just what you searched for in a few milliseconds.
Therein is the secret for Euclideon. The typical 3D Engine is the equivalent of having to travel from the beginning to the end of the Internet to find what it needs, whereas Euclideon is the equivalent of skipping 90% of it and bringing the results in milliseconds. This, in turn, makes Euclideon a highly optimized graphics engine, and the indexing of the files happens when you�re converting the high resolution polygon models into the Euclideon point-cloud data format (the equivalent of Google crawling the Internet)
The former of the two approaches uses up quite a lot of processing power with computation that is wasted because 90% of what it�s dealing with can�t be seen by the user to begin with, while Euclideon frees up that computation for other things.
Under the traditional methodologies today, the graphics cards are just sufficient enough to handle the polygon detail and all the computations, but under the Euclideon methodology, that same graphics card suddenly has nothing to do and is vastly overpowered for the engine.
So what do we do with a graphics card that is sitting around twiddling its thumbs?
Now we have room for improvement, assuming you can even imagine improving on photorealistic 3D at 25-30 FPS on the CPU alone. Stuff like astronomical resolutions for screens, ultra high fidelity, lots of GPU free to do more advanced physics and lighting calculations, and more. Instead of sharing those calculations on the GPU as an afterthought after the initial 3D calculations, now the entire card is free to unleash its full attention on it.
So the claim of Unlimited Detail is accurate, but it�s a contextual statement. Unlimited Detail, but you don�t need to load all of infinity at once to see it any more than you need to see the entire universe you live in to claim it has unlimited detail. All you can see is what is in front of you at any moment, and you take for granted that the rest of infinity exists because it�s there when you get around to seeing it.
Essentially, Euclideon engine works like the real world does. Everything is made of tiny atoms, and nothing exists except what you can see at any given moment at the level of detail you can actually see.
Polygons are a lot like a Classical Physics approach while Euclideon is the Quantum Physics approach.
Voxel Animation
Which brings us to the sticking point concerning the ability to animate point-cloud data. It�s not exactly easy, and it has come with a lot of ups and downs. Originally, it was thought to just be impossible and that was that, but we now know that�s silly because of the three points from Arthur C. Clarke at the beginning of this article. Eventually, somebody actually did manage to figure out a way to animate sparse-voxels but the conclusion was akin to the second stage of revolutionary technology statements:
Yes it can be done, but it�s not worth the effort.
With Euclideon, I�m going to say they figured out the animation aspects further and will progress into the third statement of revolutionary technology:
I said it was a good idea all along.
The people in this industry years ago who came out and said that point-cloud data was the future of the graphics industry were likely correct. The problem with being a visionary is that often times you make predictions that are completely accurate, but far ahead of their time.
download file now