Settings

Theme

Bitbucket Down - Major Outage

status.bitbucket.org

71 points by Jake232 8 years ago · 65 comments

Reader

ericfrederich 8 years ago

I've said it before and I'll say it again... It makes zero sense to centralize something that was meant to be decentralized.

We need issues, pull requests, comments, milestones, wiki, etc... all to be decentralized. No reason this stuff cannot be modeled using existing Git objects.

GitHub, GitLab and BitBucket are extremely similar. Almost 100% overlap, you could use the lowest common denominator between the 3 and you'd still have 95% of the features.

  • exikyut 8 years ago

    1. Businesses need a "halp everything is on fire" button that costs them $1000 a second but will fix EVERY issue, (virtually) guaranteed by SLA. Ultimately that is what these orgs provide; the hosting is kind of almost secondary to that.

    2. Actually building real-world implementations of "Issues, pull requests, comments, milestones, wiki, etc..." requires top-tier engineer[ing] time and energy to build and maintain. The current Git-on-steroids-aaS approach provides a very successful way to fund that undeniable cost.

  • epicide 8 years ago

    I think git-the-tool (i.e. the CLI tool) makes sense to be decentralized as it's at the core. If that tool were built around the idea of a central repository (like SVN), then you cannot escape that centralization (without using what are effectively hacks).

    I don't think it is wrong to then choose to have a centralized workflow on top of it. There are tradeoffs for either one, so choose what works best for your team.

    For example, I couldn't imagine trying to onboard new client employees to a project without having some sort of centralized single-point-of-truth repository. Getting them to grok GitHub (and friends) is hard enough, sometimes.

  • rightos 8 years ago

    > We need issues, pull requests, comments, milestones, wiki, etc... all to be decentralized. No reason this stuff cannot be modeled using existing Git objects.

    https://www.fossil-scm.org/ pretty much does this.

    • tyingq 8 years ago

      That is pretty interesting and does model those things in the same data store. But the data store isn't git objects, as far as I can tell...

      "Fossil stores its objects in a relational (SQLite) database file"

  • TomasHubelbauer 8 years ago

    What software aside from Git Appraise (decentralized code reviews) and Git Dit (decentralized issue tracking - alpha quality) do you know of that could be used to move more of what is kept in GitLab/GitHub/BitBucket databases into Git objects and Git tree in order to make Git decentralized again?

  • shakna 8 years ago

    Sounds like you might like Fossil. [0]

    I don't see git reaching beyond what it does right now, but a few others do, though they all involve a learning curve.

    [0] https://www.fossil-scm.org

  • yellowapple 8 years ago

    You could always use all three. That way, if one goes down, there are still two others ready to take its place.

  • hobofan 8 years ago

    > No reason this stuff cannot be modeled using existing Git objects.

    That's the easy part.

    The hard part is discovery and keeping stuff like teams and moderation when you decentralize it. Advancements in Blockchain-tech, IPFS and similar will make this viable soonish-ly, but I think we are not quite there yet from a technological standpoint.

    • sillysaurus3 8 years ago

      How is blockchain related to this?

      • rightos 8 years ago

        Didn't you get the memo? If you stamp enough buzzwords on everything, it magically works, regardless of their relevance to the application.

        • hobofan 8 years ago

          I see you got the memo that being dismissive of everything that contains buzzwords is still popular ;)

          Both IPFS and blockchain are definitely applicable to decentralizing applications. E.g. DSound[0], a distributed version of Soundcloud is built with IPFS + a blockchain(not necessarily a fan of STEEM though). There could just as well be something analogous for Github, Bitbucket, etc..

          [0]: https://steemit.com/music/@prc/introducing-dsound-a-decentra...

      • hobofan 8 years ago

        Blockchains allow for a distributed, mutable permissioned system that are required(?) for features like "Who is allowed to push to this named repository?".

        You can also leave stuff like that out, but then you will end up with a system where everyone can only push to their own forks leaving us with a very splintered distributed system. To compensate we would need a much better system for distributed content discovery than anything I've seen so far.

        • icebraining 8 years ago

          You don't need a Blockchain for that. You just store the public keys of the allowed users on the repository itself.

          Giving readable names to repositories is a possible use case, although I'd just use DNS.

          • hobofan 8 years ago

            Yeah, I guess you are right, a blockchain is probably not an absolute requirement for that. I think though, that decentralized maintenance of open source project would be a logical next step, and a blockchain-based system could certainly be useful there. This could help prevent a lot of stale/abandoned projects from dying where the only maintainers of a project leave it to rot and there is no one left to review PRs coming from the community.

      • jlgaddis 8 years ago

        AFAICT, blockchain is the new hammer that is the solution to all of the world's problems.

gurelkaynak 8 years ago

Now that sysadmin guy who told me: "what happens when bitbucket goes down?" when I asked him to move our repos to the cloud, he is smiling. Sometimes it's best to keep stuff in your own servers, if you have any...

  • ajbetteridge 8 years ago

    We have Bitbucket on our own servers, and we're moving it back to either Atlassian's hosting or to Github. The reason being that we spend far too much time per month hand holding the server when it goes mental and takes all of the RAM and then decides not to server any pages. And it's not for a lack of resources on the server either, 16Gb RAM and 4 core virtual machine, running Linux. So we have more downtime than either Github and Bitbucket combined.

    • gurelkaynak 8 years ago

      Any guesses why that happens? It's weird that a git server goes mental just to serve a repo. We had gitlab on my previous job and git server on my preprevious job and never ever had server down problems. This period is like 5 years total.

  • skinnymuch 8 years ago

    What about when your server and stuff goes down? Or has a bug or issue that causes downtime or similar issues? Your servers could have better uptime than Bitbucket, but maybe not.

  • beberlei 8 years ago

    not that your own servers are up 100% all the time. But the reserve in that case you could be smiling at the sysadmin guy while he scrambles for a fix, he is just happy not being at fault and having to fix it ;-)

taesu 8 years ago

Oh boy, any git down on a Monday is bad. Imagine being in their office right now.

  • AlfeG 8 years ago

    Its holiday today in Ukraine. Yet I am at work... Trying to push some code...

  • exikyut 8 years ago

    Will be interesting to hear about what happened/went wrong...

rsp1984 8 years ago

Great. I need to push some stuff real soon now. In the last 12 months Bitbucket had an uncomfortably high number of issues. But whenever I think about moving our company code to GitLab or GitHub I envision going into a world of pain with my eng. team.

Has anyone got some advice for pain-free migration to GitLab or GitHub?

  • zmb_ 8 years ago

    Git isn't centralized so there is no need to "move" the code. You can use Bitbucket, GitLab and GitHub all at the same time -- just add multiple remotes, I do this as a part of my normal setup. You can even pull directly between developer machines without the need for any centralized repository. The tricky part is all the tooling around the code (issues, wikis, CI, etc.), but no matter which provider you use for it, they will have downtime.

  • sevensor 8 years ago

    We moved over to GitLab a couple of weeks ago. We're small, so the calculus is a lot different for us. The migration basically consisted of clicking the import button and making sure everybody's remotes were changed over. We switched because we got tired of Atlassian trying to sell us all of their services all the time. The systems that make a big organization run more smoothly are a drag on a small team, and the more we saw of Atlassian's offerings the less we wanted them.

  • jazoom 8 years ago

    GitLab is way better than Bitbucket IMO but also has way too much downtime.

exikyut 8 years ago

Often there's no preservation of past failure states so people can see (for whatever reason) what the failure looked like. Here's what https://bitbucket.org/chromiumembedded/cef/ (a totally random repository that was in my history) looks right now: http://archive.is/g0I6O

qualitytime 8 years ago

First, my HDMI port blows up and have to work on a small laptop screen.

And now I can't access my repositories..

Not happy.

  • fsniper 8 years ago

    I'll up the ante, there is a hurricane in Ireland.

  • pvg 8 years ago

    And you started a lousy off-topic thread where you and others are competing to complain about downvoting and the weather.

  • exikyut 8 years ago

    Yikes. What happened to the HDMI port, out of curiosity?

    And I presume macOS? If Linux, X11 can do some interesting hacks with scaling and panning. (`xrandr --output ____ --scale` and `xrandr --output ____ --panning`)

  • qualitytime 8 years ago

    Downvote? Even more miserable.

  • quuquuquu 8 years ago

    This is why it is important to keep backups of backups ^.^

    EDIT: I upvoted you twice, HN just loves to downvote anything ;)

    • yebyen 8 years ago

      When you have to painstakingly earn the right to downvote, you just can't feel bad about using it once in a while.

      • exikyut 8 years ago

        Hi, I've always been curious what the quota/threshold is. My last account (i336_) got to 1000+ (before I accidentally locked it...) and I never saw the downvote button. Is it like 10,000 or something?

        • sevensor 8 years ago

          Either they tweak the threshold over time, or there are other factors than karma. I'm sure I gained a downvote button well before 1000. I don't think it's particularly useful -- I only use it when I think a comment isn't worth reading at all. I usually upvote people when I think they're wrong, so we can have an interesting discussion.

        • smudgymcscmudge 8 years ago

          Your user page shows you how much karma you need to hit the next level. In your case, you need 172 to downvote comments. https://news.ycombinator.com/user?id=exikyut

        • yebyen 8 years ago

          It's 500 last time I heard, but you don't get a downvote button on every comment.

          And there is no downvote button on posts (or if it is, I haven't reached that threshold yet.)

        • have_faith 8 years ago

          I think I got it around 500~. There could be a manual component to it?

Jake232OP 8 years ago

Investigating - Following reports from customers starting at 12:45 UTC, Bitbucket Cloud became unavailable. Our engineering team is currently investigating this issue. We will provide an update as soon as we have further information. Oct 16, 12:56 UTC

tyingq 8 years ago

" Our engineering team has identified the root cause of the issue and a fix has been applied. We are currently verifying that the incident is fully resolved. Oct 16, 13:48 UTC"

toyg 8 years ago

The page shows that git over https is still up, so hopefully it's just a peripheral problem rather than a fundamental one.

zitterbewegung 8 years ago

If you have one of these services you should probably invest in a backup system. Either a self hosted Gitlab or even just a clone of your repositories on a server or like AWS CodeConnect .

fazilakhtar 8 years ago

It's back for me.

rjralgar 8 years ago

"Minor*"

  • exikyut 8 years ago

    I can't access bitbucket.org, so...

    Edit: / works, repos don't

    • LeifCarrotson 8 years ago

      What does work is

          git clone https://username@bitbucket.org/organization/repo.git
      
      so you can probably still get by. Just the website is down.
      • yebyen 8 years ago

        This doesn't work for me. (Edit: and, it's moot now, because it seems to be coming back up!)

        I also have 2fa enabled (it's a private team) and my repo is private... I do seem to reach the repo, and it tells me I'll need an app-specific password to proceed.

        Too bad the interface to create one is down.

      • Peaker 8 years ago

        All the various CI systems and scripts with hard-coded SSH-based git access will be hard to change to use HTTPS for just a few hours.

      • exikyut 8 years ago

        Oh! Interesting, this is good!

    • rjralgar 8 years ago

      Sorry, sarcasm alert. They are showing "We ran into a minor glitch.". I definitely don't agree with their assessment.

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection