open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: Fixing Lag: Character Nodes
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 2 [3] 4 5

Author Topic

Yuki Kulotsuki
Posted - 2010.08.21 01:19:00 - [61]
 

Edited by: Yuki Kulotsuki on 21/08/2010 01:19:37
Originally by: CCP Atlas
freeing Jita up for important things like ... scams in local
I may just have a new signature.

Hienz Doofenshmirtz
Posted - 2010.08.21 02:33:00 - [62]
 

does this make 5 dev blogs in 5 days. stop writing blogs and get back to coding

just kidding thanks for keeping us in the loop, and writing dev blogs that almost no one will really understand. keep on trucking and keeping us informed, if you need a ghost writter for your blogs so you can keep coding, I'm looking for a new job.

ghosttr
Amarr
ARK-CORP
Intrepid Crossing
Posted - 2010.08.21 02:42:00 - [63]
 

I have a question regarding location stuffsRazz

When you are in a system does the node calculate all of the distances from you -> another object. Or does it just give all clients in system a 'broadcast' of objects?

Also are static objects (planets, gates) in the target system loaded (from the client) upon jumping, or do they have to be transmitted by the server aswell?


dakin
Minmatar
Kymmer-yn-Edeirnion
BR0WNC0ATS
Posted - 2010.08.21 04:25:00 - [64]
 

Quote:
Even though the metrics that we have gathered are for the Jita node, this change will have a positive effect on all loaded nodes in the cluster--with Jita and Fleet Fight nodes benefiting the most. The reason I use Jita here is that it has a very predictable load pattern whereas fleet fights are anything but. However, the same principles apply. Before this change you would be making something like 5-10 server calls to your location node to finish jumping, each one of these calls could take a long time to complete. Now you'll be making something like 4, with the rest returning very quickly.

We're hoping you'll be able to tell the difference the next time you decide to invade your nearest friendly neighbor.


Was this applied to Singularity? Because after today's mass test, I can say there was no improvement on jumping.

Tres Farmer
Gallente Federation Intelligence Service
Posted - 2010.08.21 04:28:00 - [65]
 

Edited by: Tres Farmer on 21/08/2010 05:53:14
Originally by: CCP Atlas
This sort of change will speed up utility functions that don't impact your solar system directly. Your client should seem a bit 'spiffier' when talking to the server. You won't see an fps increase but you don't have to wait as long for things like loading up the map, right clicking on other players and things of that nature. There is also a bit less for the location node to do so it has more buffer for the pew-pew.

Yeah.. map is way more speedy than it was - VERY GOOD work there.
If the mail just could handle as well?! I just cleared the cache of my mail and it seems a bit faster now.. why isn't this 'clear the cache' an automated process when closing down the client?

And why does it take +30seconds for the mail window to be responsive after opening it the first time in the actual client session?
This just looks broken! What is it waiting for? New mails? Why can't it show the old mails.. all that had been cached already and should be saved locally on my hdd right away with the menu on the left?
And then if new ones come in have it show them..
Making me wait for this is not user friendly!

And if I start the client.. why can't this mail demon load NEW mails right away in the background from the server.. along with all the other info I need at startup. Make it an option in the ESC menu to let my 'preload' my mails upon starting the client.
I really hate it having to wait for the mail windows response.. I really really do.


On a more related note.. what comprises the services the location node is handling actually? Would you be able to give us a list of stuff this node handles pretty please?

And yeah.. best blog of the lot so far!

Tres Farmer
Gallente Federation Intelligence Service
Posted - 2010.08.21 04:39:00 - [66]
 

Edited by: Tres Farmer on 21/08/2010 05:24:24
Originally by: CCP Atlas
Originally by: Mashie Saldana
With these new nodes, would it be possible to have dedicated AI nodes to bring Sleeper AI to all NPCs in EVE?

Actually, the NPC AI is a perfect example of a system that needs to live on the location node. However, there are not any outstanding scalability issue with that.

There are no technical reasons Sleeper AI or something akin to that isn't on more or all NPC's, it's a game mechanical / balancing issue which is outside my expertise... ugh

Why does the NPC AI need to live on the location node? What services do (and need) to live on the location node anyways.. hm, I'm repeating myself here.. Rolling Eyes

What kind of pvp/spaceship/space related services run on a location node?
How much of the load on a location node does those services cause?
Why do these services need to run on the same node as the NPC AI or the inventory system or local chat?

Siiee
Recycled Heroes
Posted - 2010.08.21 05:15:00 - [67]
 

Originally by: Tres Farmer
Why does the NPC AI need to live on the location node? What services do (and need) to live on the location node anyways.. hm, I'm repeating myself here.. Rolling Eyes


If I'm understanding it right you can think of the location node as the "local" node. You can only interact (pew pew) with NPCs that are on grid with you at your location (and the grid is attached to your solar system instance), and thus it has to be on the location node.

Tres Farmer
Gallente Federation Intelligence Service
Posted - 2010.08.21 05:31:00 - [68]
 

Edited by: Tres Farmer on 21/08/2010 05:34:53
Originally by: Siiee
Originally by: Tres Farmer
Why does the NPC AI need to live on the location node? What services do (and need) to live on the location node anyways.. hm, I'm repeating myself here.. Rolling Eyes

If I'm understanding it right you can think of the location node as the "local" node. You can only interact (pew pew) with NPCs that are on grid with you at your location (and the grid is attached to your solar system instance), and thus it has to be on the location node.

I can interact (pew pew) with other players who are NOT running their thought-processes on that local node.
The reason we got so dumb NPC might be just because the sleeper AI uses x-times more cpu and is only bearable in w-space where you don't got so many PvE'ers as in k-space. You see where I'm going with this?!
Let's say CCP manages to 'outsource' NPC AI onto other node(s). Want more intelligent NPC? Just install more NPC nodes..

I can't argue on facts here as my knowledge of the server and the processes are so tiny. So any insight from Atlas is highly appreciated.

So again.. a table with services running on the local node and their load (might even need diff cases listed like Jita/PvE/PvP) would be needed for us all to look at..

Nareg Maxence
Gallente
Posted - 2010.08.21 05:34:00 - [69]
 

Originally by: Tres Farmer
Why does the NPC AI need to live on the location node? What services do (and need) to live on the location node anyways.. hm, I'm repeating myself here.. Rolling Eyes

The AI needs to control the NPCs which are part of the space simulation. It needs do decide whether the NPC flies up or down or left or right or shoots at you or warps away. Anything that's part of the space simulation needs to be on the location node.

I guess it is technically possible that you could treat NPCs as a ships controlled by a computer player, just like we have our ships and we control them via the client. The question is what that would accomplish. In low load systems with people just ratting or mining or whatever, there wouldn't be much point in trying to offload the probably fairly low contribution to load that the AI accounts for. In high load systems where there are fleet fights or system like jita, people aren't interacting with the AI, so it's idle anyway.

The only situation where I could see a possible benefit, would be in mission hubs, but then you will have to replace the direct interaction of the AI with a message passing system to get the commands from the AI node to the location node and back. Thinking of the amount of NPCs that spawn and despawn all the time, it seems to me that this would increase network trafic on the node by an unacceptable amount, and would there then be a benefit at all?

Tres Farmer
Gallente Federation Intelligence Service
Posted - 2010.08.21 05:45:00 - [70]
 

Edited by: Tres Farmer on 21/08/2010 05:45:39
Originally by: Nareg Maxence
Originally by: Tres Farmer
Why does the NPC AI need to live on the location node? What services do (and need) to live on the location node anyways.. hm, I'm repeating myself here.. Rolling Eyes

The AI needs to control the NPCs which are part of the space simulation. It needs do decide whether the NPC flies up or down or left or right or shoots at you or warps away. Anything that's part of the space simulation needs to be on the location node.

I guess it is technically possible that you could treat NPCs as a ships controlled by a computer player, just like we have our ships and we control them via the client. The question is what that would accomplish. In low load systems with people just ratting or mining or whatever, there wouldn't be much point in trying to offload the probably fairly low contribution to load that the AI accounts for.

Low load is never the reason to something, that's right.

Originally by: Nareg Maxence
In high load systems where there are fleet fights or system like jita, people aren't interacting with the AI, so it's idle anyway.

Well it doesn't need to be idle then, does it? At the moment PvE in high load systems isn't worth it because of the problems it causes..
What was there first - the egg or the chuck?

Btw.. we don't have orbiting planets/moons/stations because of the load they cause on the server.. ever thought of a way around this?
Have a Orbital-Bodies-Node and let it do the calcs needed for all the stuff that's supposed to be moving. If a player enters system his client talks to this node and get's up to date information about all the orbiting bodies and their status. We should have orbiting planets. No?

Originally by: Nareg Maxence
The only situation where I could see a possible benefit, would be in mission hubs, but then you will have to replace the direct interaction of the AI with a message passing system to get the commands from the AI node to the location node and back. Thinking of the amount of NPCs that spawn and despawn all the time, it seems to me that this would increase network traffic on the node by an unacceptable amount, and would there then be a benefit at all?

I have no fricking clue because I have no idea about the services running on the location node nor the load each of the services cause under all those cases.. do you?

Siiee
Recycled Heroes
Posted - 2010.08.21 05:54:00 - [71]
 

Originally by: Tres Farmer
I can interact (pew pew) with other players who are NOT running their thought-processes on that local node.



The "thought process" is irrelevant (and absolutely trivial, even Sleeper AI is still fairly basic from my experience with it, only barely more "intelligent" and more random).

Normal NPC AI tends to boil down to something like this (bolded the important part)
--
if(ships_on_grid)
shootAt(first_ship_on_grid)
--

selecting a target from a list, not difficult, but that information of the first ship on the grid is only available in the solar system simulation, and so has to occur on the location node. If you were to run NPC AI on a separate node it would need to have all of that grid information available from the location node, and you'd end up having to maintain two separate copies of the same information, lots of additional overhead, not worth it.

They would be better off working on breaking the location node away from a single Sol (think a grid-sized location) which would accomplish a lot more in the grand scheme of things.

Tres Farmer
Gallente Federation Intelligence Service
Posted - 2010.08.21 06:14:00 - [72]
 

Edited by: Tres Farmer on 21/08/2010 09:58:24
Edited by: Tres Farmer on 21/08/2010 06:26:33
Originally by: Siiee
Originally by: Tres Farmer
I can interact (pew pew) with other players who are NOT running their thought-processes on that local node.


The "thought process" is irrelevant (and absolutely trivial, even Sleeper AI is still fairly basic from my experience with it, only barely more "intelligent" and more random).

Normal NPC AI tends to boil down to something like this (bolded the important part)
--
if(ships_on_grid)
shootAt(first_ship_on_grid)
--


We'll who here would like more intelligent NPC?
Ok, who here does have an idea or know if this is possible with the current implementation of things.. like NPC AI on the location node?

Originally by: Siiee
selecting a target from a list, not difficult, but that information of the first ship on the grid is only available in the solar system simulation, and so has to occur on the location node. If you were to run NPC AI on a separate node it would need to have all of that grid information available from the location node, and you'd end up having to maintain two separate copies of the same information, lots of additional overhead, not worth it.

How much information is that? The position of ships on the local grid? Wow.. that's really a big chunk of information there.. especially as the AI would only have use for the player position as it would know it's own very well and the servers could trust the NPC AI and their position broadcasts.

Look.. the space-simulation-service does all the calcs for physics, the weapons, the movements, the cap, the damage.. all that buzz..
NPC AI doesn't need access to that data.
Players havn't access to that data either for other ships on the grid.
Why should NPC need that?

Originally by: Siiee
They would be better off working on breaking the location node away from a single Sol (think a grid-sized location) which would accomplish a lot more in the grand scheme of things.

and how do you streamline things?
Do grids with POS or Custom Office need a NPC AI service idling around?

Let's think of a process that controls a spawn of NPC. Like 2 BS, 2 Cruiser, 4 Frigs.. all they need is YOUR position and your shiptype.
Then they decide what kind of tactic they will use against you and start it. No?
And that service runs on a dedicated node which runs many others of those processes.

If the NPC could get more intelligent we wouldn't need hordes of them anymore, resembling a slaughtering when fighting against them and have this big gap between PvE and PvP in fittings/tactics.

Same AI/node could be used for the incarna NPC.

Who knows.. with enough resources freed up we might even witness running missions on jumpgates again.
Man.. how I miss that.
I was so excited as a noob when I could see someone fighting some NPC in a mission while I was approaching the gate.
Good, I admit these days we wouldn't take much notice with WTZ.. ugh

Jita Dancer
Posted - 2010.08.21 09:58:00 - [73]
 

"Indeed. What is on our roadmap is in fact to allow for non-destructive (e.g. not kick everyone out) live remapping of solar systems. I don't know when this will be a reality but it's definitely something that we are very interested in.

With such a system in place when a solar system you're in gets too loaded to play nice with the other solar systems the load balancer would kick in automatically and you would just pause for a bit and then continue as if nothing had happened on a spiffy new node."

Why dont you go the other way? When a node gets busy, it remaps ALL THE NON BUSY SYSTEMS somewhere else? Perhaps not as beneficial as a busy node moving itself, but by definition - the non-busy nodes are either empty or have a very small number of players attached and will (more) likely be able to successfully hand those players off onto a different node, leaving the bulk of players not exposed to a "high load handover"... Just a thought.

Dragonia Redtail
Posted - 2010.08.21 10:32:00 - [74]
 

Can CCP please explain to me how we are going to know when and where a fleet fight takes place?

In case of killing a pos, yes its known what we attack and where. But ehm, a 250 man roaming gang is not easy to announce in advance.

Is there not an option to be found to get a node to assist the other node when it gets up to 90% load and starts trowing up?

After all, you know we all love this game, but the lag monster involved is our biggest target....

Mashie Saldana
Minmatar
Veto Corp
Posted - 2010.08.21 11:34:00 - [75]
 

Originally by: CCP Atlas
Originally by: Mashie Saldana
With these new nodes, would it be possible to have dedicated AI nodes to bring Sleeper AI to all NPCs in EVE?

Actually, the NPC AI is a perfect example of a system that needs to live on the location node. However, there are not any outstanding scalability issue with that.

There are no technical reasons Sleeper AI or something akin to that isn't on more or all NPC's, it's a game mechanical / balancing issue which is outside my expertise... ugh

That is interesting so server load isn't a reason to keep the current (near nonexistant) NPC AI?

Kolatha
Posted - 2010.08.21 11:44:00 - [76]
 

Originally by: Jita Dancer

Why dont you go the other way? When a node gets busy, it remaps ALL THE NON BUSY SYSTEMS somewhere else? Perhaps not as beneficial as a busy node moving itself, but by definition - the non-busy nodes are either empty or have a very small number of players attached and will (more) likely be able to successfully hand those players off onto a different node, leaving the bulk of players not exposed to a "high load handover"... Just a thought.


I would think this would be the better option. When a system starts getting busy it means something is going down. If you move that busy system with all the participants ramping up their game you are just asking for trouble, unless you can do it smoothly and seamlessly. Moving the busy node also means you need to make sure you have sufficient idle nodes waiting just for this purpose.

On the other hand I can see how a number of large roving gangs could cause some pretty hefty internal bandwidth usage as they cause system after system to get hit with remapping.

CCP Atlas

Posted - 2010.08.21 11:50:00 - [77]
 

Originally by: Kolatha
Originally by: Jita Dancer

Why dont you go the other way? When a node gets busy, it remaps ALL THE NON BUSY SYSTEMS somewhere else? Perhaps not as beneficial as a busy node moving itself, but by definition - the non-busy nodes are either empty or have a very small number of players attached and will (more) likely be able to successfully hand those players off onto a different node, leaving the bulk of players not exposed to a "high load handover"... Just a thought.


I would think this would be the better option. When a system starts getting busy it means something is going down. If you move that busy system with all the participants ramping up their game you are just asking for trouble, unless you can do it smoothly and seamlessly. Moving the busy node also means you need to make sure you have sufficient idle nodes waiting just for this purpose.

On the other hand I can see how a number of large roving gangs could cause some pretty hefty internal bandwidth usage as they cause system after system to get hit with remapping.


Yes, indeed. Today we do either, depending on the situation. Sometimes the system itself is moved and sometimes the other systems, leaving the fight alone.

CCP Atlas

Posted - 2010.08.21 11:57:00 - [78]
 

Originally by: Tres Farmer
...
And why does it take +30seconds for the mail window to be responsive after opening it the first time in the actual client session?
This just looks broken! What is it waiting for? New mails? Why can't it show the old mails.. all that had been cached already and should be saved locally on my hdd right away with the menu on the left?
And then if new ones come in have it show them..
Making me wait for this is not user friendly!
...

This doesn't sound like the way it should work. Can you submit a bug report for us? Mention me please so that I get it.

Selene D'Celeste
Caldari
The D'Celeste Trading Company
ISK Six
Posted - 2010.08.21 13:19:00 - [79]
 

Edited by: Selene D''Celeste on 21/08/2010 13:19:28
Originally by: CCP Atlas
Originally by: Tres Farmer
...
And why does it take +30seconds for the mail window to be responsive after opening it the first time in the actual client session?
This just looks broken! What is it waiting for? New mails? Why can't it show the old mails.. all that had been cached already and should be saved locally on my hdd right away with the menu on the left?
And then if new ones come in have it show them..
Making me wait for this is not user friendly!
...

This doesn't sound like the way it should work. Can you submit a bug report for us? Mention me please so that I get it.


I believe this happens if you haven't cleaned out your evemail cache for a long time. When I remove all old mails from trash, etc, the mail system loads much faster. Over time as mails accumulate it slows down.

Also thanks for the blog. It's good to see logical, common sense improvements being made to the server topology. If you guys keep at it we should be zooming around again in a few more months =D

Originally by: CCP Atlas
Market hubs like Jita have the potential for load balancing stations separately of the solar system and other stations. That is something we are currently investigating as a possible 'end-all' fix to Jita. There is a fair bit of game design involved and I'm not making any promises however. :-)


This would be another nice improvement along the same line of thought. Glad to see that you're looking into it.

Louis deGuerre
Gallente
Malevolence.
Posted - 2010.08.21 16:35:00 - [80]
 

Excellent blog, keep em coming Razz

Caladain Barton
Navy of Xoc
The Remnant Legion
Posted - 2010.08.21 18:58:00 - [81]
 

Edited by: Caladain Barton on 21/08/2010 18:58:55
CCP, we were having 1000 man battles on un-reinforced nodes without them crashing. you could squeeze 2k into a reinforced node back before dominion.

Just so you know how high the bar *was* at. Are you planning on restoring the game to that level, or to the 500 man battle on regular node, and 1000 on the special nodes?

Luke S
Zeta Corp.
Posted - 2010.08.21 20:11:00 - [82]
 

I got a little off topic question and maybe thinking a little too far ahead.
Have you guys thought about the servers for Dust 514? Will they be going through XBox live servers/ PlayStation network? or will you have dedicated servers for your MMOFPS. How will they be talking to each other.

If this is too early, Stick this post in your Dev blog list. I like to hear how it would work.

d4shing
Posted - 2010.08.21 23:05:00 - [83]
 

Edited by: d4shing on 21/08/2010 23:07:58
Edit: whoops, accidentally posted blank message...

Anyways, I wanted to suggest that you guys use some sort of algorithms to pick which nodes to RF without being told by players.

I mean, you have timers for all these ihubs and POSes and SBUs coming out of reinforced, right?

Surely no more than 5ish ihub ref cycles come up on a given day... why not just automatically RF those nodes?

Seems like it would be pretty easy to do...

Tres Farmer
Gallente Federation Intelligence Service
Posted - 2010.08.22 02:43:00 - [84]
 

Edited by: Tres Farmer on 22/08/2010 02:48:11
Originally by: CCP Atlas
Originally by: Tres Farmer
...
And why does it take +30seconds for the mail window to be responsive after opening it the first time in the actual client session?
This just looks broken! What is it waiting for? New mails? Why can't it show the old mails.. all that had been cached already and should be saved locally on my hdd right away with the menu on the left?
And then if new ones come in have it show them..
Making me wait for this is not user friendly!
...

This doesn't sound like the way it should work. Can you submit a bug report for us? Mention me please so that I get it.

Bug Report ID is 99609. Good luck!
[Edit] I cleaned the cache of my mail 24hours ago I think.. but, yeah.. never before that afaik. But to be honest, I don't do this with thunderbird either and that one gets WAY more mails on a daily basis.


Too bad I couldn't get an answer on my question for more detail about those services/processes running on those location node(s). Sad
Or is there a blog coming in about those details? Would love it!

Bagehi
Association of Commonwealth Enterprises
Posted - 2010.08.22 07:31:00 - [85]
 

Originally by: Tres Farmer
Edited by: Tres Farmer on 22/08/2010 02:48:11
Originally by: CCP Atlas
Originally by: Tres Farmer
...
And why does it take +30seconds for the mail window to be responsive after opening it the first time in the actual client session?
This just looks broken! What is it waiting for? New mails? Why can't it show the old mails.. all that had been cached already and should be saved locally on my hdd right away with the menu on the left?
And then if new ones come in have it show them..
Making me wait for this is not user friendly!
...

This doesn't sound like the way it should work. Can you submit a bug report for us? Mention me please so that I get it.

Bug Report ID is 99609. Good luck!
[Edit] I cleaned the cache of my mail 24hours ago I think.. but, yeah.. never before that afaik. But to be honest, I don't do this with thunderbird either and that one gets WAY more mails on a daily basis.


Too bad I couldn't get an answer on my question for more detail about those services/processes running on those location node(s). Sad
Or is there a blog coming in about those details? Would love it!


My mailbox also takes quite a while to load up. Usually, when I first login, I'll open that, then go about doing other things (checking skills, checking my isk, checking my orders, check current fleets, etc) while waiting for the mail to come up. I just loaded up the client and timed it. Just over 18 seconds from opening the mail window to the mail loading up.

Cheap Dude
Posted - 2010.08.22 09:10:00 - [86]
 

Nice read.. Got one question though. When you warp to a location you end up in making a new grid or joining an existing one. Can't grids be splitup like the other stuff you mentioned in the blog? This way only the grid lags when numbers get too high and not the whole solarsystem.

Mielono
Caldari
SWARTA
Posted - 2010.08.22 14:09:00 - [87]
 

I think they probably could but then there would be a session change/load screen setup in system instead of the seamless warp that we have at the moment. So they could probably isolate the different areas in the solar system onto different nodes but then possibly introduce lag into people warping into a area in the solar system instead of just the people jumping into the system. Right now the lag monster seems to be on the side of the defender, introducing that without revamping how the system handles instant large loads and fleet fights would probably just end up hurting fleet fights more than it already does.

The idea itself is not without merit though, if the fight really starts to heat up at a certain location on the map, somehow figuring out how to bubble that area off onto another core while the rest of the system continues on as business as usual would be nice and would allow smaller split off fights to occur without being troubled by the lag monster.

ElfeGER
Deep Core Mining Inc.
Posted - 2010.08.22 16:49:00 - [88]
 

Originally by: CCP Atlas
Originally by: James Bryant
Hey guys,

Fantastic dev blog. Certainly answers a whole slew of architectural questions that have been lingering in my head for some time.

My question is in regards to location node load balancing. The very fact that fleet fight requests are necessary seems to indicate a lack (or possibly not enough) automated load balancing of location nodes.

No doubt this is not a new idea to you all, so I am wondering what the difficulties are in implementing the capability to offload light traffic location nodes to underutilized CPUs when heavy traffic nodes start to throttle the CPU. Is there not a way to manage the connection while the handoff is being made? Or is it more an issue of detection and implementing the proper hysteresis in the algorithm (so that nodes don't start swapping around CPUs needlessly)?



Indeed. What is on our roadmap is in fact to allow for non-destructive (e.g. not kick everyone out) live remapping of solar systems. I don't know when this will be a reality but it's definitely something that we are very interested in.

With such a system in place when a solar system you're in gets too loaded to play nice with the other solar systems the load balancer would kick in automatically and you would just pause for a bit and then continue as if nothing had happened on a spiffy new node.


it might be easier to move 100 systems with 5 people each compared to moving a system with 500 people
closing down empty systems on the loaded node might help as well so that they restart somewhere else when needed or right away

Akita T
Caldari Navy Volunteer Task Force
Posted - 2010.08.23 00:05:00 - [89]
 

So, when can we expect to see problematic "location nodes" getting moved to a different CPU core in real-time (with a brief interruption in service before that happens, but without a server reboot) ?!?

SFX Bladerunner
Minmatar
Aperture Science inc.
Posted - 2010.08.23 01:16:00 - [90]
 

Originally by: Akita T
So, when can we expect to see problematic "location nodes" getting moved to a different CPU core in real-time (with a brief interruption in service before that happens, but without a server reboot) ?!?


Do I really have to say it?...

It starts with an S, and ends with a


Pages: 1 2 [3] 4 5

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only