open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: CarbonIO and BlueNet: Next Level Network Technology
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 [2] 3 4

Author Topic

lisaaa
Posted - 2011.06.20 17:48:00 - [31]
 

Originally by: CCP Veritas

Quote:

CarbonIO has not been turned on for the client as yet, but since the client doesn't sling packets for a living that's not a big deal. It will need to be activated before BlueNet is leveraged at all, so you can expect it to happen before the multi-player Incarna release.



Quote:


CarbonIO has been live on TQ cluster-wide for a week.






I don't get >.<

Flynn Fetladral
Royal Order of Security Specialists
Posted - 2011.06.20 17:52:00 - [32]
 

Great blog! Will be nice to see multi threaded client coming to a computer near you sometime in the future.

Callic Veratar
Posted - 2011.06.20 17:53:00 - [33]
 

Originally by: Bagehi
Graphs. So pretty. This leaves me wondering if someone, somewhere in the near future is going to rewrite the GIL so that it can send processes to other cores/CPUs to be run, rather than keep them all on the same core. But... I don't know much of anything about coding so, to whoever explains why this is a dumb idea, be nice please.



If I understood what was explained, the GIL is a transactional lock. Processing that require it's function must be done in the order they were received to prevent memory trampling. Think of it like this: I buy a ship, then I sell a stack of minerals. My bank balance should be [balance] = [balance] - [ship] + [minerals]. But, if you tried the transactions at the same time you could end up with [balance] = [balance] - [ship] with your minerals missing or [balance] = [balance] + minerals] with a free ship.

CCP Curt

Posted - 2011.06.20 17:55:00 - [34]
 

BAH. defaulted to my throwaway alt. That last post was meant to be by CCP Curt

lisaaa
Posted - 2011.06.20 18:00:00 - [35]
 


What did Veritas mean by that the CarbonIO isn't turned on for the client, but it has been live on TQ for a week now ???

Callic Veratar
Posted - 2011.06.20 18:04:00 - [36]
 

Originally by: lisaaa

What did Veritas mean by that the CarbonIO isn't turned on for the client, but it has been live on TQ for a week now ???


Exactly that, it's enabled on the server but not on the client yet. It doesn't need to be on both ends to work.

CCP Curt

Posted - 2011.06.20 18:07:00 - [37]
 

Originally by: lisaaa

What did Veritas mean by that the CarbonIO isn't turned on for the client, but it has been live on TQ for a week now ???


He meant the TQ cluster, server-side. The client will see no benefit from this since it spends almost no time sending packets. We have not fully tested all the client systems with it yet, but will turn it on once we have. So far so good.

Originally by: Miss Modus
What was happening at the four peaks of the CPU% per user graph where the CarbonIO spiked much higher than the original StacklessIO?


Egads I was hoping that would slide by. I almost erased them out but thought it best not to fudge the data in any way.

These graphs are from the initial 24-hour deployment to a fully-loaded pair of proxy nodes on TQ. The "spikes" are the result of a spinlock-bug I fixed the next day. Essentially, since the communications is now multithreaded, its possible for one thread to call for the close of a socket at the same time another one is writing data to it. This condition is guarded against with a mutex of course, but if it occured at EXACTLY the same time (multi-core) there was a "hole" in the logic that spun a core until the socket was actually released. This had no effect on throughput, but would tie down a core until the TCP/IP stack finnaly came back and said "yup he's gone".

Spinlocks on mutli-core systems are more efficient in cases of small hold times and infrequent contention,
but in this case I needed to use a more conventional sleep/event type lock.

Now if you were really clever you'd ask about those little peaks that grow slowly over time toward the end of the run ;)

Originally by: Uncanny Valley
Is BlueNet going to be available for the Incarna release, or just CarbonIO?
When are you going to "publish" BlueNet data?

This was created to service the data requirements of Incarna, but in so doing, the door was opened for other systems to use it. CarbonIO was created so that something like BlueNet could function (ie- send/receive packets off the GIL). BlueNet is now available for internal teams to start examining and see if they can leverage.

Originally by: Ishina Fel
Whoa, wait a moment! Where did that one come from? I heard no mention of this in any prior devblogs, and not in the fanfest coverage either Surprised And yet, this is the kind of thing people will frantically gobble up, the kind of thing everyone wants to read.

When are which parts of this getting enabled? Is that an official feature of the Incarna expansion, or just something that happened to be ready at the same time? What are you hoping to do with this technology in the future?


Well.. the simple version is we didn't know if this would work at all, let alone gain any efficiency. CarbonIO was written to open up a data-path through MachoNet that was wide enough to allow Incarna data to flow without hurting cluster performance, and we can show that it does that. Other than that, we didn't really expect any efficiency gains. But apparently the ground-up rework with this new paradigm *did* result in some gains.. some big ones.. but we only discovered that once we ran it, and then only over the last week and a half or so.

Something this big and basic being rolled into TQ, we never thought it would work the first time, that's why we took it so slow to begin with. But.. it DID work the first time, and fast, surprising everyone (me most of all).

So to sum up, we didn't tell anyone because we didn't know if we would have anything to tell, and even if we did, we didn't know when it would be available.

Set the way-back machine to 2 weeks ago with me huddled at my desk praying to the gods of no-crashy as this was rolled this into Jita for the first time. Hoping that whatever went wrong I'd be able to identify and fix before Incarna had to ship.

Yeah you didn't want to be me. "reproduction steps: get 2000 users to perform trade actions randomly on Jita.. ".

But I'll be damned if it didn't work, and lowered CPU by 10%

tatsudoshi I
Gallente
The Venus Project - Zeitgeist Movement
Posted - 2011.06.20 18:07:00 - [38]
 

Edited by: tatsudoshi I on 20/06/2011 18:09:15
Originally by: CCP Curt
Now if you were really clever you'd ask about those little peaks that grow slowly over time toward the end of the run ;)



I actually thought that was what MissModus was asking about.. So what are the ever growing spikes and when or if do they stop?

CCP Curt

Posted - 2011.06.20 18:14:00 - [39]
 

Originally by: tatsudoshi I
Edited by: tatsudoshi I on 20/06/2011 18:09:15
Originally by: CCP Curt
Now if you were really clever you'd ask about those little peaks that grow slowly over time toward the end of the run ;)



I actually thought that was what MissModus was asking about.. So what are the ever growing spikes and when or if do they stop?


They are stat crunching. The longer the server runs the more data it sifts through when polled for information. Toward the end of the run it actually starts to be visible. CarbonIO manages idle time quite a bit differently (not necessarily better, just different) than StacklessIO so while those spikes were previously lost in the noise, they became visible.

This is being addressed now to more efficiently deal with that data.

Motriek
Selective Pressure
Rote Kapelle
Posted - 2011.06.20 18:16:00 - [40]
 

Please clarify what portions of the server stack this impacts and benefits. As understand it, there are proxy servers, sol nodes / servers, and database servers. Does this primarily benefit the proxies or the sols?

CCP Curt

Posted - 2011.06.20 18:23:00 - [41]
 

Originally by: Lykouleon
CCP CURT,
01010111011010010110110001101100
00100000011110010110111101110101
00100000011011010110000101110010
01110010011110010010000001101101
0110010100111111

Seriously nice blog and some great info. And enough graphs to appease my graph-throne <3



cat nerd.cpp
#include <stdio.h>

//--------------------------------------------------------------------
int main( int argn, char *argv[] )
{
char in[]="01010111011010010110110001101100"
"00100000011110010110111101110101"
"00100000011011010110000101110010"
"01110010011110010010000001101101"
"0110010100111111";

int pos = 0;
while( in[pos] )
{
char accumulator = 0;
for( int i=0; i<4 && in[pos] ; i++ )
{
accumulator += in[pos] == '1' ? 1<<i : 0;
pos++;
}

printf( "[%d:%c]:", (int)accumulator, accumulator );
}

printf( "\n" );
return 0;
}

$ gcc -o nerd nerd.cpp ;./nerd
[10:
]:[14:]:[6:]:[9: ]:[6:]:[3:]:
[6:]:[3:]:[4:]:[0:]:[14:]:[9: ]:[6:]:
[15:]:[14:]:[10:
]:[4:]:[0:]:[6:]:[11:
]:[6:]:[8]:[14:]:
[4:]:[14:]:[4:]:[14:]:[9: ]:[4:]:[0:]:[6:]:[11:

]:[6:]:[10:
]:[12:
]:[15:]:


I don't get it.

-Curt

Sakura Ren Fenikkusu
Posted - 2011.06.20 18:29:00 - [42]
 

Edited by: Sakura Ren Fenikkusu on 20/06/2011 18:29:01
I definetly love to read dev blogs and articles like this.

I'm a hobbyist/indie developer, working on a few small games, and trying to learn everything I can, so that in the future I can make something as great as Eve Online.

It is one thing to read articles about how things could be done, and how they have been done in the past, another to see what live MMOs are doing to make their games better from a technical standpoint.

Keep up the good work.

EDIT: CCP Kurt broke the forums....

CCP Warlock

Posted - 2011.06.20 18:30:00 - [43]
 

Originally by: CCP Curt

//------------------------------------------------------------------------------
int main( int argn, char *argv[] )
{
char in[]="01010111011010010110110001101100"
"00100000011110010110111101110101"
"00100000011011010110000101110010"
"01110010011110010010000001101101"
"0110010100111111";

int pos = 0;
while( in[pos] )
{
char accumulator = 0;
for( int i=0; i<4 && in[pos] ; i++ )
{
accumulator += in[pos] == '1' ? 1<<i : 0;
pos++;
}

printf( "[%d:%c]:", (int)accumulator, accumulator );
}

printf( "\n" );
return 0;
}

$ gcc -o nerd nerd.cpp ;./nerd
[10:
]:[14:]:[6:]:[9: ]:[6:]:[3:]:[6:]:
[3:]:[4:]:[0:]:[14:]:[9: ]:[6:]:[15:]:
[14:]:[10:
]:[4:]:[0:]:[6:]:[11:
]:[6:]:[8]:[14:]:[4:]:
[14:]:[4:]:[14:]:[9: ]:[4:]:[0:]:[6:]:[11:

]:[6:]:[10:
]:[12:
]:[15:]:


I don't get it.

-Curt



It says "Will you marry me?" - I think you forgot that its 8 bits per character...

Ambo
I've Got Nothing
Posted - 2011.06.20 18:30:00 - [44]
 

Originally by: CCP Curt

I don't get it.

-Curt



http://www.roubaixinteractive.com/PlayGround/Binary_Conversion/Binary_To_Text.asp

Paste binary, click 'to text' Wink


Also, awesome dev blog and AWESOME work.

tatsudoshi I
Gallente
The Venus Project - Zeitgeist Movement
Posted - 2011.06.20 18:33:00 - [45]
 

Edited by: tatsudoshi I on 20/06/2011 18:36:38
Originally by: CCP Curt

I don't get it.

-Curt



Drink more or less coffee, which ever is the opposite of what you are doing at this very moment! (Read with over-exited speaker voice)

EDIT, Ambo love your toon!

Bienator II
Posted - 2011.06.20 18:39:00 - [46]
 

thats a lot of effort to avoid touching the GIL. (Python was probably a bad longterm choice for the server logic but i guess you have good reasons why you can't migrate)

have you experimented with Jython (http://www.jython.org)? It is basically fully multithreaded python on the JVM but was between 0.1x and 5x slower (single threaded comparison) the last time i played with it. It is under active development and gets faster with every release.

So that might get interesting in future if you want to run on 32+ cores which isn't that uncommon for a java server any more :)

CCP Curt

Posted - 2011.06.20 18:45:00 - [47]
 

Originally by: CCP Warlock

It says "Will you marry me?" - I think you forgot that its 8 bits per character...


<mirage>[curt|/home/curt]$ cat nerd.cpp
#include <stdio.h>

//--------------------------------------------------------------------
int main( int argn, char *argv[] )
{
char in[]="01010111011010010110110001101100"
"00100000011110010110111101110101"
"00100000011011010110000101110010"
"01110010011110010010000001101101"
"0110010100111111";

int pos = 0;
while( in[pos] )
{
char accumulator = 0;
for( int i=0; i<8 && in[pos] ; i++ )
{
accumulator += in[pos] == '1' ? 1<<(7 - i) : 0;
pos++;
}

printf( "%c", accumulator );
}

printf( "\n" );
return 0;
}

<mirage>[curt|/home/curt]$ gcc -o nerd nerd.cpp ;./nerd
Will you marry me?


Ah no, sorry married with 3 kids already.

Coffee? heck no I go right to the source- Diet Coke.



Mike deVoid
Firebird Squadron
Terra-Incognita
Posted - 2011.06.20 18:46:00 - [48]
 

Always worth telling CCP employees and CCP teams when they do impressive work, to balance when I post in the rage threads.

GJ guys :).

CCP Curt

Posted - 2011.06.20 18:47:00 - [49]
 

Originally by: Bienator II
thats a lot of effort to avoid touching the GIL. (Python was probably a bad longterm choice for the server logic but i guess you have good reasons why you can't migrate)

have you experimented with Jython (http://www.jython.org)? It is basically fully multithreaded python on the JVM but was between 0.1x and 5x slower (single threaded comparison) the last time i played with it. It is under active development and gets faster with every release.

So that might get interesting in future if you want to run on 32+ cores which isn't that uncommon for a java server any more :)


I know that it gets examined from time to time along with other possible solutions, but that's way outside my paygrade. I just make the little box go *ping*

Korerin Mayul
Amarr
Posted - 2011.06.20 19:02:00 - [50]
 

i like the smell of this.
i hope this stuff gets some air-time with the 'high level languages in supercomputing' crowd.

Herr Nerdstrom
Caldari
Havoc Violence and Chaos
Merciless.
Posted - 2011.06.20 19:05:00 - [51]
 

Edited by: Herr Nerdstrom on 20/06/2011 19:07:43
I have a couple questions:

- What is the percentage of operations that are affected by this offloading of GIL and non-GIL related code, and that therefore benefit from these changes? Optimizing code to reduce CPU usage of that code by 35% is great, but it's not that much of a benefit if that code only accounts for 3% of the load.
- It sounds like a lot of the efforts here are basically to reinvent nonblocking I/O, and perhaps splitting the network I/O (including compression and encryption) off to another thread, with the inclusion of completion ports. What am I missing?
- The final graph showed the difference between StacklessIO and CarbonIO CPU usage. However, the CPU being examined was never at 100% to begin with. Without testing these upgrades on a core that was maxed, i.e., during a fleet battle, how can we tell how much actual benefit will be produced? Providing stats of a core that was already functioning fine, and therefore without significant lag(TM), doesn't really tell us anything about the effect of the upgrades. If the changes will bring the CPU usage of a 1000 pilot fleet battle to something less than 100%, then I think CCP can say something about the effectiveness of the changes.

I still think this is CCP spending years trying to overcome the limitations of a 'convenient' language like Python. The better choice, in my opinion, still would have been to use a non-interpreted and fully-featured language like C or C++.

That being said, I appreciate the efforts of the various teams at CCP, and applaud their progress. This dev blog was also excellent in detail and content, and I didn't catch a single grammatical or spelling error...please keep them coming.

CCP Curt

Posted - 2011.06.20 19:27:00 - [52]
 

Originally by: Herr Nerdstrom
Edited by: Herr Nerdstrom on 20/06/2011 19:07:43
I have a couple questions:

- What is the percentage of operations that are affected by this offloading of GIL and non-GIL related code, and that therefore benefit from these changes? Optimizing code to reduce CPU usage of that code by 35% is great, but it's not that much of a benefit if that code only accounts for 3% of the load.



Sort of, but it can depend greatly on how you measure and define "load". I'm not going to p lay games, but it bears mentioning that if I define "load" as "time spent processing on a CPU" and that 3% is spent doing inefficient things that cause the other 97% to spin in circles, then clearly by eliminating it the overall efficiency gain can greatly surpass 3%.

I do think there was some of that going on, with the GIL being requested too often, the overall system efficiency went down, CarbonIO requests it less.

But lets take that point at face value- clearly it is not the case since the graphs show a very substantial gain, on the order of 50%, for proxies. So comm must have been taking up at least 50% of the load, yes? And most likely far more since reducing the entire comm library to a single NOP would probably not work.

Quote:

- It sounds like a lot of the efforts here are basically to reinvent nonblocking I/O, and perhaps splitting the network I/O (including compression and encryption) off to another thread, with the inclusion of completion ports. What am I missing?

Not re-invent.. redeploy.. StacklessIO was using completion ports and nonblocking I/O, but it was tightly coupled to the GIL. ie- there was built in seriality. On a whiteboard or flowchart it would be a simple matter of erasing some lines and penciling in "a miracle occurs" and *poof* decoupled, but in the real world it took a long development effort to remove.

The end effect is that not JUST communications occurs off-GIL (as StacklessIO did) but that the code USING it can ALSO occur off-GIL.

Quote:

- The final graph showed the difference between StacklessIO and CarbonIO CPU usage. However, the CPU being examined was never at 100% to begin with. Without testing these upgrades on a core that was maxed, i.e., during a fleet battle, how can we tell how much actual benefit will be produced?


You make a good point- if both graphs cross 100% at 1000 users, then their shape is irrelevant. If that shape conferred no information, but they do. Load growth follows fairly linearly at ever load we have tested live, and the labratory tests (where we pushed big-iron blades to 100% routinely) predicted savings.

So far the lab data has been borne out on live loads, which can be taken as evidence that other lab predicitons are likely to be true. We can't be sure until its actually tried.. and we are standing by.. but a reasonable conclusion based on all available data is that there will be some savings on near-100% loads.

Which is irrelevant!

CarbonIO was never about reducing load directly, that's a side effect! BlueNet is about reducing serial load, when teams like Gridlock start using it to go-wide on the flying-in-space systems.

tatsudoshi I
Gallente
The Venus Project - Zeitgeist Movement
Posted - 2011.06.20 19:35:00 - [53]
 

Edited by: tatsudoshi I on 20/06/2011 19:35:37
Originally by: CCP Curt
Coffee? heck no I go right to the source- Diet Coke.


I have heard around the health-heads that diet drinks on long term stores "something" in the bone marrow or something. Because there is no energy in the drink the body does not "see" the drink as something that needs to be digested so it does not know what to do and so it stores it.. or something. I have not researched this for health-fanaticism so use as seen fit. Only FYI. I have switched to coffee as I can't drink liters of it as I could with soda.

pmchem
Minmatar
GoonWaffe
Goonswarm Federation
Posted - 2011.06.20 19:35:00 - [54]
 

CarbonIO sounds great (nice work). If I may redirect a little bit, any chance you guys will be able to use a MPI implementation for python (or... any underlying server code) anytime soon for taking _real_ advantage of having multicore CPUs?

This would be the ultimate solution for server-side scaling of fleet fights, as you're probably already aware.

Chaotic Alacrity
Posted - 2011.06.20 19:50:00 - [55]
 

Seriously, half the reason I play this game is so that I have context for the awesome devblogs. Great job.

Are the internal discussions regarding new optimizations made possible by CarbonIO centered on reorganizing the calls to existing modules or are people starting to look for ways that they could offload logic currently in python to external libraries (that can now run concurrently)?

Klandi
Consortium of stella Technologies
Posted - 2011.06.20 20:32:00 - [56]
 

Great post - lots of detail and juicy bits Laughing

Looking forward to lots more - as much or as deep as you want to go

Sri Nova
Posted - 2011.06.20 20:38:00 - [57]
 

im surprised the developers of python have not come over there, to give all of you a good beating. For abusing their environment in ways that they could never have imagined.

Its amazing what ccp has pulled off with this engine and its fascinating reading about your accomplishments .
im sure a lot of head banging, hair pulling, and sending off dear friends to the loony bin occurred during all this .

I bet all of you are elated when you make gains like this .and no words can adequately commend you guys for the hard work you put in to this .

but awesome job guys really its appreciated especially when we are playing the game, enjoying it and not realizing that its your hard work enabling us immerse ourselves in your universe so easily .

Glyken Touchon
Gallente
Independent Alchemists
Posted - 2011.06.20 20:41:00 - [58]
 

Great job!

Originally by: CCP Curt
Egads I was hoping that would slide by. I almost erased them out but thought it best not to fudge the data in any way.

These graphs are from the initial 24-hour deployment to a fully-loaded pair of proxy nodes on TQ. The "spikes" are the result of a spinlock-bug I fixed the next day.


Good choice. Real data with an explanation comes across much better with this audience.

Salene Gralois
K-2
Posted - 2011.06.20 21:29:00 - [59]
 

That'll cure me of my geek-itch for a week.
Thanks, i loved this type of blog in the past and (surprise :D) i still love them.

Jim Luc
Caldari
Rule of Five
Vera Cruz Alliance
Posted - 2011.06.20 22:13:00 - [60]
 

So would this allow for a better server-side physX model in the flying-in-space portion of Eve? Also, the ability for twitch-based controls (fast frigates and fighters need the sensation of actually flying around with roll and yaw, and strafe left/right without pointing the nose in a new direction), and fleet formations?


Pages: 1 [2] 3 4

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only