open All Channels
seplocked EVE Information Portal
blankseplocked New Dev Blog: New Forum Security Blog - Cookie Derp
 
This thread is older than 90 days and has been locked due to inactivity.


 
Pages: 1 2 3 [4] 5 6 7 8 9

Author Topic

Misanth
RABBLE RABBLE RABBLE
Posted - 2011.04.12 04:04:00 - [91]
 

Originally by: Kuroki Meisa Kennedy
Originally by: mazzilliu
apparently the guy who first reported the issue and later got banned said that his initial exploit report was incomplete, but there was no ccp effort to get him to elaborate.

perhaps it would be an improvement to have some sort of followup for security related reports, in case the reporter does not understand how to properly demonstrate an exploit, to try to get him to communicate clearly, rather than brush them off as another incomplete bug report or potential troll. i think if that happened the forums might have gone down some time sooner.


+1 I also feel just killing the messenger is wrong and makes the P in CCP stand for police.


Communication is not CCP's strongest side, never has been. They misunderstand players, we misunderstand them. They present stuff in a way that aggravates the playerbase, even tho it could easily be made in a more appealing fashion. Etc.

I gave up petitioning CCP for now, last couple of petitions they just gave me standard replies asking for information I already submitted in the first petition. It is no point raising your voice when you talk with the deaf.

Johanna Tychi
Posted - 2011.04.12 04:52:00 - [92]
 

OMG, CCP will get the most secure forums in the whole internets if we keep trying to break them. CCP could sell that tech later ;)

Keep on rolling!

Jo

Tasko Pal
Aliastra
Posted - 2011.04.12 04:54:00 - [93]
 

As I see it, this is going to foreshadow the Incarna failure this summer. The causes of failure will be somewhat different. Here, the decisions made were pretty bizarre. I can see how this security flaw might have slipped past, but what boggles me is the abandonment of something like eight years of content and discussion. Among other things, it means that CCP went with a completely different system from front to back, when at a glance, they apparently just wanted to modify the front end look and some functionality. I think CCP might find with hindsight that a completely new system would be more likely to have the sort of security flaws which were uncovered, while a system that kept their old backend database, probably wouldn't have these problems.

I think the Incarna failure will be due to massive overselling of the first generation of Incarna content. It's something like two months to the release and we still have no concrete discussion of any serious game-related excuse for me to get out of my pod. I doubt any release in the past few years which has issued significant new content (incursions, exploration, level 5 missions, epic arcs, wormholes, etc) has been that vague about what was being provided so soon before the launch. Sure, the new content sounds interesting, but how many years will it take to integrate it so that we can use it for something other than a fancy variation on station spinning? Maybe CCP should start talking about that so that expectations meet the actual level of content.

As I see it, big missteps in a forum upgrade probably indicate deeper problems in CCP with new content generation. I'm a one-game man. Eve really has some powerful and compelling content which other games currently can't touch. But this summer looks pretty weak for new content. At least, the BFF group and related parties are visibly improving the existing Eve experience. And the war on lag works.

Vult
Posted - 2011.04.12 05:28:00 - [94]
 

Originally by: Johanna Tychi


Keep on trolling!

Jo


Fixt.

Makko Gray
Pheno-Tech Industries
Crimson Wings.
Posted - 2011.04.12 05:36:00 - [95]
 

I find it interesting that HTML could be injected into a signature but not script and really hope you do not look at the use of the script tag as the only indicator of script injection as it can also occur on HTML attributes and through various encoding techniques (some browser specific).

There are some good XSS cheat sheets you can check against for testing cross site scripting vectors such as this one: http://ha.ckers.org/xss.html

Also .NET & C# rocks as long as you know what your doing (ASP.NET WebForms is a bit sucky though).

DTson Gauur
Caldari
Blend.
Nulli Tertius
Posted - 2011.04.12 05:45:00 - [96]
 

Decent blog, and I know you're not the guy to answer this Sreegs but...

You're using a GPL'd (GPLv2 license actually) software, so have you guys actually read the license and understand what it means?

I hereby request the modified sourcecode as is my right within GPL license.

Ball is in your park now CCP, obey the license.

Gnulpie
Minmatar
Miner Tech
Posted - 2011.04.12 05:55:00 - [97]
 

How on earth was it possible that these holes weren't discovered by QA? Amazing!

And where are the people responsible for that desaster anyway? Where are the people responsible for the new forums?

However CCP Sreegs is the security guy here and it looks like he is doing a great job and working his ass off right now to get things done and to keep the community up to date. And that is GREAT!

However, the security department should have been involved before the release of the forums? Was it?



Bargealta McSpacebuxx
That's What a Spy Would Say
Goonswarm Federation
Posted - 2011.04.12 05:56:00 - [98]
 

So are you going to post how the exploit worked after it's fixed for the curious, or no?

Originally by: DTson Gauur
You're using a GPL'd (GPLv2 license actually) software, so have you guys actually read the license and understand what it means?

Pretty sure GPLv2 still has that web service loophole that basically lets you ignore it for web-hosted apps.

Londo Cebb
Official Market Discussions Troll
Posted - 2011.04.12 05:59:00 - [99]
 

Thank you for this explanation of the situation so far.

I was rather ****ed off when I found out the extent of the problems with the new forums, and still am.

I have lost a fair amount of my faith in your company to keep my data secure, but your formal apology and acknowledgement of the problems has restored some small amount. I think even you will admit that you still have a long way to go to earn back that trust.

I am looking forward to a follow up blog detailing exactly what went wrong (to the extent that you can).

I would like to thank you again for owning up to your mistakes. That is the first step in making sure something like this never happens again.

Yuki Kulotsuki
Posted - 2011.04.12 06:01:00 - [100]
 

Originally by: Bargealta McSpacebuxx
Originally by: DTson Gauur
You're using a GPL'd (GPLv2 license actually) software, so have you guys actually read the license and understand what it means?

Pretty sure GPLv2 still has that web service loophole that basically lets you ignore it for web-hosted apps.
Pretty much this. The software is not being distributed to you and thus you are not entitled to the source.

Mara Rinn
Posted - 2011.04.12 06:16:00 - [101]
 

Thanks for the update, CCP Sreegs. The new forums have many more problems besides the security flaws though. It's not your department, I know, but perhaps there's some way you can influence the release process so that QA and process stakeholders such as CSM can work together to control the release of non-security patches?

Of course, this involves upgrading CSM from "chicken" class stakeholders in the process to "pig" class stakeholders in the process. We're where the money comes from, after all.

A little change to the process so that Singularity always contains the next release candidate. This could work in CCPs favor to reduce the number of bugs that occur due to unintended interactions between a publicly tested feature and an internally tested bug fix.

From your dev blog, the insinuation is that the player who got banned didn't actually tell you how to reproduce the problem - is this the message you intended to present?

And for DTson gauur - according to the GPL, a developer only has to release the source code for software that they have given to the customer in some form. Thus if CCP had sold us forum software to put on our machines, or had sold us a box containing the forum software, we'd be entitled to the source code too. Since CCP haven't delivered us a software product (they're renting us a software service), the source availability is a non-issue.

I'm sure CCP will do the right thing and contribute changes back to the YAF codebase, where those changes apply to other users. I don't know what happens behind the scenes, but I can't think of many instances where one login can post as multiple different identities. It doesn't make sense in the greater world of forum software.

And for folks complaining about Akita T raving about the forums being stinking pile of dog excrement: remember what feedback CCP gave about Akita T's Technetium complaint? No you don't, because there wasn't any. I think it's pretty clear what Akita T is "messaging" by raving about the new forums at every opportunity.

But GPL and forum UX flaws are not the topic of this thread. The topic of this thread is how awesome CCP Sreeg's metaphorical beard is, and how certain security troll's beards are lacking in the non-neck department. Keep it up CCP.

Herschel Yamamoto
Agent-Orange
Nabaal Syndicate
Posted - 2011.04.12 06:16:00 - [102]
 

Originally by: Misanth
Originally by: Dacil Arandur
of all people Akita T has the most to lose!


No. The playerbase as a whole, has, if that monster of a forum comes back.


Better than this stinky old turd. Not nearly as much better as it ought to be, but better nonetheless.

Originally by: Bargealta McSpacebuxx
So are you going to post how the exploit worked after it's fixed for the curious, or no?


It was ******ed simple. Basically, there were two completely unrelated problems. One was HTML injection via signatures - basically, HTML was blocked for the post body, but you could put whatever you wanted in your sig via a fairly simple workaround. It started when people wanted to use font colours and images in their sigs(functionality not yet implemented on the new forum), and then they realized it wasn't limited to making sigs colourful.

The second problem was even dumber. The forum's method of telling what character you're posting as was a simple cleartext string in the cookie, of the type "lastSelectedCharacter=1840703239". Fair enough - it's a non-crazy way to remember which of your three toons to post as - except that the server just took the character ID and trusted it completely, with no checking. If I set my last ID to CCP Sreegs(which is the ID number I used above - they're easy to look up), then I could post as Sreegs, edit Sreegs' posts, and have access to all mod tools and hidden forums Sreegs can see...without ever having to actually log in to Sreegs' account. Just set your ID in the cookie, the server takes it as gospel without checking, and you're in as anyone you like.

Seriously, I can't overstate just how ******ed this was. "Derp" is far too weak a word.

Gnulpie
Minmatar
Miner Tech
Posted - 2011.04.12 06:17:00 - [103]
 

Originally by: DTson Gauur
You're using a GPL'd (GPLv2 license actually) software, so have you guys actually read the license and understand what it means?


GPL2 §0:
Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted...


As long as they only RUN the program and don't distribute it, the GPL2 gives you ****.

Yuki Kulotsuki
Posted - 2011.04.12 06:24:00 - [104]
 

Originally by: Mara Rinn
Of course, this involves upgrading CSM from "chicken" class stakeholders in the process to "pig" class stakeholders in the process. We're where the money comes from, after all.
So mittens is king of the piggies?

Florestan Bronstein
24th Imperial Crusade
Posted - 2011.04.12 06:26:00 - [105]
 

Edited by: Florestan Bronstein on 12/04/2011 07:34:36
Quote:
This takes me to vulnerability reporting, which has played an interesting role in this whole process. If you are aware of or discover a vulnerability in one of our systems you are encouraged to send an email to security@ccpgames.com, file a petition and/or a bug report. If you do this there are two items which are paramount to us having the information we need to respond properly.

The widespread perception in the shc thread was that the only way to get CCP to act fast is to publicize and exploit the vulnerabilities.

CCP has a long history of being extremely slow to react to exploits in other parts of the game (even exploits that are not just "bugs" such as the Monkeysphere incident).

One frequently voiced concern was that the new forums must be taken down before the weekend or else the risk of "serious" exploits getting into the wild would become too high.

Are you convinced that the forums would have been shut down on Friday/Saturday if you had just received a couple of bug reports?

At what times is security@ccpgames.com monitored? What's the maximum time you might not be aware of a vulnerability that has been forwarded to that mailbox?

Do you really have the internal leverage to shut down central parts of the website when the exploit has not yet been observed in the wild?

There were some reports that parts of the exploits were applicable to EVE Gate - I remember someone mentioning that it is possible to write evemails from unsubscribed accounts etc - was EVE Gate not taken down because these claims are simply not true or because EVE Gate is a too central/integrated part of the EVE experience (and cost/benefit did not justify shutting it down)?

Some people claimed part of the vulnerabilities had already been mentioned on forums and/or reported to CCP during the public testing of the new forums. Can you comment on that?


Whether javascript injection was possible or not seems to be mainly a question of how well you sanitize HTML attributes, there are countless places you can stick a bit of js code you want to have executed and most of them don't involve any "<script>" tag.
An attacker would probably also look to encode his javascript in some way to mask it & protect it from sanitization (pretty likely that any larger site includes a js framework that already offers support for the decoding of bas64, hex, ...).
For an attack to be successful it doesn't have to be standards compliant - stuff like background-image:url('javascript:alert("Hi")') will for example work with IE 6 and shouldn't work with more modern browsers - but that can already be bad enough...


edit: maybe CCP could propose some sort of Incident Response/Vulnerability Disclosure agreement with its community?
I am thinking of a commitment to reply to a reproducible & security-related bug report within less than x hours/days, to provide a workaround/fix within y days and the assurance that a detailed description of the vulnerability with attribution to whichever players had a central role in reporting it to CCP/making it reproducible for CCP will be made available once the fix has been deployed.
This could help foster an atmosphere of "responsible disclosure" (lol).

Othran
Brutor Tribe
Posted - 2011.04.12 06:32:00 - [106]
 

Apologies if this has been directly answered - I have looked and don't see it.

CCP Sreegs - did the security team test the new forums for common vulnerabilities before they went live?

I'd expect every company to pen test internet-facing assets prior to any deployment to a live environment. I'm just curious who, if anyone, was responsible for testing in this instance.

Skex Relbore
Gallente
Red Federation
RvB - RED Federation
Posted - 2011.04.12 06:34:00 - [107]
 

I'm still wondering who thought it would be a good idea to roll the new forums out on a Friday?

Jimmae
Posted - 2011.04.12 06:41:00 - [108]
 

Fun fact: EveGate was vulnerable to XSS too during closed beta. One should think the Web Devs had been made aware of possible security issues back then already.

Next time have proper penetration testing done by outsiders! How can we hold you to your own standards if you aren't?

Also: Educate your personnel!

I am a Software Engineer myself and while I knew about SQL Injections and Path Traversal, I had no clue about MANY other things like XSS, XSRF or XEE.
I accompanied a few penetration tests and they made me realize that most issues could be avoided during the development process already by sensitizing programmers BEFORE they get kicking.

token guy
Posted - 2011.04.12 06:51:00 - [109]
 

CCP Sreeeeeeeeeeeeegs made a goodpost.

What's going on here?

Jimmae
Posted - 2011.04.12 06:53:00 - [110]
 

Originally by: Othran
I'd expect every company to pen test internet-facing assets prior to any deployment to a live environment. I'm just curious who, if anyone, was responsible for testing in this instance.


I absolutely aggree with you. However, unless you have a dedicated in-house specialist you should have it done by professionals. There are many more attack vectors than just the OWASP top 10 and new ones constantly emerging.

My company has been working with n.runs and they sure know their stuff. They found some reflective XSS which I would have found too. The main issue though we would have NEVER found during in-house testing and it could have lead to a compromise of our complete server infrastructure.

Chribba
Otherworld Enterprises
Otherworld Empire
Posted - 2011.04.12 07:34:00 - [111]
 

Let's just hope there is no PIE incident!

Keep up the good work!

/c

Ban Doga
Posted - 2011.04.12 07:38:00 - [112]
 

Edited by: Ban Doga on 12/04/2011 07:38:59
Originally by: Ban Doga
Edited by: Ban Doga on 11/04/2011 09:53:23
Originally by: Bomberlocks
We'll see what Sreegs posts in his blog, but I'm not entirely convinced that CCP will be honest as to the extent of the problem as I think it might open them up to possible legal problems.


The blog will reiterate the statements already made.
This will include "injection of HTML", "user data was not at risk" and "security's job is to react to issues - not to prevent them by reading code".
It will contain a more lengthy and (slightly) more detailed explanation of "What" happened but not "Why".

Questions regarding "Why" will be met with "Policy says 'No'", "I already explained that", "I say what I said" and "Asking about bans or warnings could get you a ban or warning yourself".


And I'll be delighted to be wrong...

Not too disappointed I wasn't wrong.

I find it a bit odd that you cast away that "the method by which your information would have been at risk would have been in the form of malware, session theft or keylogging of your local machine" with "it's always a best practice to keep your computers safe".
If that's the stance on security could you please get someone to get rid of the "You are leaving CCP-land. Evil people might be trying to attack your computer." warning when following links in the forum. It's the same thing and you said "I'm stating outright that customer data was never at risk.".
So I guess we don't need that warning...


I'm also wondering about your two example mails to report vulnerabilities.
None of them state "I will continue" or "I will stop", yet you seem to imply the first one will continue but the second won't.
What's the magic word/phrase/indicator here?


Do you also agreed that one has to make actual use of a (potential) exploit at least once to confirm it is there?

Toshiro GreyHawk
Posted - 2011.04.12 07:42:00 - [113]
 

Originally by: Jimmae
Originally by: Othran
I'd expect every company to pen test internet-facing assets prior to any deployment to a live environment. I'm just curious who, if anyone, was responsible for testing in this instance.


I absolutely aggree with you. However, unless you have a dedicated in-house specialist you should have it done by professionals. There are many more attack vectors than just the OWASP top 10 and new ones constantly emerging.

My company has been working with n.runs and they sure know their stuff. They found some reflective XSS which I would have found too. The main issue though we would have NEVER found during in-house testing and it could have lead to a compromise of our complete server infrastructure.



Yeah ... if you don't have in house people who just eat, sleep and breathe this stuff then they are going to be behind the curve. Most places aren't big enough or don't have their priorities set high enough for this kind of thing.

The thing is - you've got people out there who do eat, sleep and breathe this stuff that're the ones trying to break into your systems.

And - it never ends.


.

Louis deGuerre
Gallente
Malevolence.
Posted - 2011.04.12 07:50:00 - [114]
 

Report security exploit, no matter how annoyingly, and get banned ?
You should give the guy a medal.
For shame CCP ugh


Madcapnl
The Rising Stars
The Volition Cult
Posted - 2011.04.12 07:58:00 - [115]
 

Kudos to you, CCP Sreegs. You are the first CCP employee to actually communicate to the community.
You have clearly explained what happened, you have admitted that stuff went belly up and that you are working to fix that. Basically that is what the community expects: just tell them what is happening and why you are doing what you are doing.

I truly think that the ano's nerf in nullsec would have landed a lot better with the community if it had been communicated sooner (or at all for that matter) and with the actual reasons, not with the made up reasons that are laughable to the community. Nerfing stuff can benefit the game, but it HURTS a lot of people. Clear and honest communication is key when some of the community is hit by a nerf. By not communicating with the community, CCP is basically saying "we know better and you are stupid, so shut up" to the commmunity. And that just pis$es them off.

Anyway, keep up the good work. Treat us like the serious people we are and we will love you long time.Laughing

Bagdon
GoonWaffe
Goonswarm Federation
Posted - 2011.04.12 08:01:00 - [116]
 

Originally by: Akita T

This was just a more visible symptom of a much deeper seated problem at CCP, namely that stuff HAS to be rushed to meet an unrealistic deadline, and damn the consequences, because the people responsible for the money are getting too antsy.



While the overall truth of rushing features to a deadline might be true, this case isn't a symptom of that. Rushing to a deadline means cutting corners, cutting features and skipping QA. The migration of posts from the old forum is probably a symptom of deadlineitis. In the case of impersonating others on the forum (I haven't bothered researching the signature thing, since it's a smaller problem) there is no way a competent developer would make that implementation choice even with a deadline gun pointed to his head. It's not a matter of QA either, since these kinds of problems are not testable. It's a simple matter of incompetence or inexperience.

Ban Doga
Posted - 2011.04.12 08:12:00 - [117]
 

Edited by: Ban Doga on 12/04/2011 08:17:38
Originally by: Bagdon
While the overall truth of rushing features to a deadline might be true, this case isn't a symptom of that. Rushing to a deadline means cutting corners, cutting features and skipping QA. The migration of posts from the old forum is probably a symptom of deadlineitis. In the case of impersonating others on the forum (I haven't bothered researching the signature thing, since it's a smaller problem) there is no way a competent developer would make that implementation choice even with a deadline gun pointed to his head. It's not a matter of QA either, since these kinds of problems are not testable. It's a simple matter of incompetence or inexperience.


I can assure you there are people out there making a living testing exactly that and nothing else.
It is possible to test this and it is possible to find that.

The sad truth is we will never know why this happened.
It might be a case of skipped QA to hold a deadline.
It might be a case of bad QA to hold a deadline.
It might be a case of a failed process (QA was scheduled but no one checked if it actually did occur).
It might be a case of incompetence ("We don't need to test that - it'll be fine").
It might be a case of ignorance (no one thought about testing that).

We will never know because none of those possibilities will make CCP look any better.
In fact if we knew what happened a lot of people would be infuriated even more ("How could you..." - yeah, I'm one of those...).

*EDIT*
And the reason people get upset even more is because of the implication you see.
"If you don't test that what else aren't you testing?"
"If your process fails there, where else aren't you keeping things on track?"
"If you ignore this, what else aren't you seeing?"

That's the real damage here - not that the forums went down for a day or that the development team is probably pretty much devastated at this point and quite some people probably busy finding out what happened in the first place. (This is all bad too, of course)

Moron78
Blueprint Haus
Posted - 2011.04.12 08:16:00 - [118]
 

I know crap about computers, forums and all. So for me, as a regular customer, this is an issue of trust. Since CCPs notice on the forums was rather scarce when the forum thing hit I did what I usually do for stuff CCP is being coy about, Kugu and Scrapheap. (Well, the latter no more.) Now as I said I don’t know much about this, but what was shown to be done on SHC seemed really basic. Which I gather to be the gist of the discussion thread over on Kugu also.

And CCP let it past. Sreegs, you say in the blog that you haven’t been able to get in scripts that would run malware, key loggers and stuff of those forums. But you are by no means sure. And you even very inelegantly try to put it on the end-used. (“Even were someone able to have injected script the method by which your information would have been at risk would have been in the form of malware, session theft or keylogging of your local machine rather than some window into our secure environment.”) Now to not get off track, I recognise that only to a very limited degree can CCP be responsible for what takes place at my computer. But, CCP made a forum where you are not ruling out that keyloggers etc could be embedded, and by leaving a rather obvious security hole for the computer savvy.

And regardless of that as I understand it the hole is by no means insignificant to CCP as it enabled reading of all subforums, including subforums where stuff potentially under NDA could be discussed. (I’m assuming that NDA information may be discussed in the closed CMS forum.)

So, my question. If CCP let this slip by what else have you not been able to catch? I hope that this is a case of Sreegs’ department being bypassed in internal processes. But as Sreegs says, they aren’t about to tell us. My issue with leaving this undisclosed is that I no longer have any trust in CCP when it comes to security measures. Why Sreegs would I trust you to rectify and make a secure forum- or anything else – when the last attempt potentially could have exposed end users and opened NDA information to the world?

Gavjack Bunk
Gallente
Genos Occidere
HYDRA RELOADED
Posted - 2011.04.12 08:17:00 - [119]
 

I love it when a Dev takes his turn in the barrel.

Sreeg's Barrel. It's CCP's answer to Schrodinger's Cat. Do we know what state he's in right now?

Psihius
Caldari
Anarchist Dawn
U N K N O W N
Posted - 2011.04.12 08:19:00 - [120]
 

Dear CCP!

As a web developer (PHP & MySQL based + all the stuff surrounding) for over 7 years now, and one of the leading once in our small country.
I should point out that your WEB team should be re-evaluated, because this is unacceptable error. It's not just a bug or small glitch - it's a huge black hole in the application security. I work myself with finances and the security it top priority, and frankly, I just don't get it how such a big mistake can be made?

Microsoft has a history with ASP & ASP.net and probably now same with the .NET - it was easy and it requires more a team of trained monkeys than a real programmers to do the job and todo real stuff you need people passionate about the tech and knowing it inside out. Not to mention there are sometimes just amazingly stupid bugs in .NET.
And definitely there should be some security guy witch knows things and is able to teach others.


Pages: 1 2 3 [4] 5 6 7 8 9

This thread is older than 90 days and has been locked due to inactivity.


 


The new forums are live

Please adjust your bookmarks to https://forums.eveonline.com

These forums are archived and read-only