Doing random things over at http://musteat.org
211 stories
·
4 followers

Fresh pcaps, free for the asking

2 Shares

One thing network people hate is having to get access to a machine (or groups of them) in order to run tools like tcpdump. They want a packet trace from somewhere, and having to obtain not just ssh access but sufficient powers to open a raw socket and start sniffing (typically root-level) is a pain. It should be unsurprising that someone eventually wrote a tool which solved the problem for them.

How did it work? Well, the machines in question (all of them) were already running some "agent" software which logged interesting things about the network activity of each host. It had root powers on the machines, and took requests from the network. Adding the packet trace capability was just a matter of extending what was already there.

The code was written, it was attached to the service, and it was shipped with little fanfare. It just kind of hung around after that point, doing its job while presumably reducing some of the more annoying angles for the humans.

Then, one Friday afternoon, rather late, someone decided it was time to post one of those feel-good announcements about what they had accomplished. This was quite a while after it had shipped, but maybe performance reviews were coming up? Nobody's really sure why it was announced at that point, or why it was late on a Friday, but that's what happened.

A few minutes later, one of those people who looks at everything and thinks "is that safe, or could someone hurt themselves or others with it" saw the post and the wheels started turning. Instead of going and digging in to the code (which would require pulling out the laptop and getting back online), they stayed on their little mobile app and replied to the post, instead. They asked some questions: who has access to this, and how is that enforced, exactly?

The hope was to get an answer back like "the network team, specifically anyone in LDAP group X" for the first part, and "we use a (tech) (tech) in the (tech) (tech) part of (tech) to verify their identity". However, unlike what happens with Star Trek scripts, the "tech" items here would actually be appropriate and solid for the task at hand. They'd look even smarter and people would feel good about the whole thing: a tool that saves time and doesn't open more holes!

The answer which came back, to put it charitably, fell short.

"We went and looked... turns out... everyone."

They had added the method to the service but hadn't added any authentication to determine who was making the request, or authorization to make sure they had the right to make it. As a result, anyone could switch it on and get tasty, tasty packet traces from any box in the fleet.

All you needed to do was send a properly-formatted network packet, and it would do the work for you. Want to see what's happening on a sensitive system that handles money, or HR data, or really, anything else run on these machines? Turn on a packet trace and hope they "forgot" to encrypt their traffic in flight.

As you probably guessed, this was unacceptable, and so, that Friday evening, the team which produced the service, the security responders, and that engineer who tends to look for the holes in things didn't get to knock off work for the weekend. Instead they had to figure out how to turn it off (even if that meant breaking things which relied on it), get that pushed out everywhere ASAP, and then see if they can leave it like that until Monday, or if they have to actually solve for the identity and permissions stuff right then and there.

This also meant people had to go digging to (try to) see if it had already been exploited. Not everyone waits for the announcement. Some people just go around thinking "hmm, what's already listening on the network and has interesting powers and curious-sounding RPC methods?", and those people would be way ahead of you.

Finally, there's the human angle to this. Obviously, a SEV was opened for this. The "user impact" field was written to be something like this by the engineer who discovered it:

Any host running (tech) since (commit hash) which was released (month) (year) would have provided packet dumps to anyone who asked politely via (tech).

Someone (who had worked on the service) couldn't leave it alone. They had to edit it. They added this:

It is perceived to have security risks associated with such data export.

In other words, "someone annoying thought this is a big deal and that's why we're working late tonight and into the weekend".

Side note: the code change which introduced this had a comment from one person saying "you know this might be a bad idea, right?"... but it was committed and shipped anyway. Oh, random commenter, if only they had listened to you.

Read the whole story
denubis
23 days ago
reply
Sydney, Australia
smarkwell
23 days ago
reply
Share this story
Delete

Engineers without borders, silos, and vendor walls

1 Share

I've seen a bit of variety in how different companies handle engineering problems. Some are big. Others are small. Some build things, and others don't. For some places, the tech is the product, but at others, it's a distraction at best. This is just to say that no one solution is expected to solve every problem, but it's still possible to find things that generally work better than others.

Let's talk about communication and teamwork. If you think those are generally good to have in your organization, you might be prone to make decisions to maintain what you already have and perhaps grow more. Keep that in mind as I describe some scenarios.

Scenario number one is a company which has a product which then depends on a bunch of different services. The product isn't important - it could involve cat pictures, influencing elections, or delivering pizzas. We'll say they have hard dependencies on 20 different services which must all be up and running for the business to function. Maybe one service is a database, another one is a cache, a third one is a group of web servers, a fourth tracks "upvotes", "likes" or "stars", and so on.

In this organization, the 20 services are owned by 20 teams in a 1:1 mapping. Team A owns service A, team B owns service B, and so on down the line. Some of these teams are quite good at what they do, and rarely cause trouble for the business. We'll even say that most of them are pretty solid, and generally don't break things.

There are, however, three services which have significant challenges. They fall over a lot. They stop working. They take down the whole operation. It doesn't matter how reliable the other 17 parts are, since one of these "terrible three" will show up and wreck the whole thing.

Fortunately, this company is enlightened, and has embraced a philosophy where people are able to talk about these things openly, and reflect on the problems and not the people involved. More to the point, the best engineers from other parts of the company are able, willing, and allowed to "wander around" into the troubled parts to help out. This kind of knowledge transfer happens, and pretty soon the roughest spots have been smoothed out, and now everything's a lot better. The cat pictures keep going out, the elections keep being hacked, and the pizzas show up on time. Life's pretty good.

Now let's jump into scenario number two. It's another company that has similar hard dependencies. They also have 20 services which must be up and running to make their business "go". Those 20 services are also run by 20 separate teams, as before, with one team owning one service. Most of them are also pretty good, and only a few of them have issues.

Unfortunately, that's the end of the similarities. Company number two has this "siloing" thing going on. If you're not familiar with the term, consider yourself lucky. It's when a company isolates one thing (a system, department, etc) from another. Well, this company has a LOT of siloing happening. People are very possessive of their services, and their team's "turf". Outsiders are to be suspected, since they probably have some nefarious intent in mind. The fact that all of these people nominally work for the same company with the same supposed goal in mind doesn't matter.

The corporate firewalls are up, they're numerous, and they're pretty tough. If a good person is on one side of the wall, and there's an interesting problem on the other that they could fix, the wall is going to prevent it. There will not be any cross-pollination between the teams. The best you might see is a few individuals moving around between teams in a shared silo, but you'll never see anything more than that.

The organization has been set up with defenses against this. People who try to poke their nose in, even if they have the best possible goals in mind, are the enemy. They must be blocked or reported. They definitely must be thwarted. They must not see the weaknesses in the system because it will be used to attack somehow and take over part of the turf!

Imagine running a company this way, where you can get these protected realms where the employees are "untouchable" and also somehow unable to deliver on the same requirements as everyone else. Worse still, you find yourself unable to send in helpers since the immune system will flare up and attack them in every way imaginable.

In such an environment, I'd expect it to become obvious by way of the general sense of malaise, acceptance of the status quo, and a lot of shrugs - "it's always been like that". "They're protected". "Those requirements don't apply to them". Maybe you've heard some of these before.

I'd hope that most people would try to avoid this with their companies, and part of that would involve breaking down the walls between groups which aren't mixing it up enough. The whole notion of engineers moving around and sharing their talents with whoever needs it has to be supported and rewarded.

Now I'm going to throw another complication into the mix with scenario three. This is a third company which amazingly also relies on 20 different services staying up and available the whole time for their business to work.

But, unlike company #1 and company #2, this one has a completely different approach to services. By default, they go out to Hacker News, find the shiniest thing in that space, and then pull out their checkbooks and throw money at whatever products they find on this romp through the web. They end up with 20 different services, sure, but they are coming from 20 different vendors.

Maybe most of the vendors are reasonably competent and so you never notice problems with them. They take your money and you get some level of service in return. It's good enough and so you probably forget about them. This is akin to the 17 or so good ones at the other two companies.

There's always a catch, though. Three of these services are garbage, just like at the other two companies. Their vendors are just not good at doing whatever they are purportedly selling. They're consuming lots of customer money and are delivering fatal errors, timeouts, or worse.

When this kind of thing happens at company 1, there's always the option of retasking some of the better engineers to do a field trip to the troubled teams to bail them out. If it happens at company 2, the silo walls prevent it, but those silo walls can in theory be torn down, and then they can turn into company 1.

But company 3? That's a different kind of trouble. All of their services are on the other side of a vendor relationship. Maybe some of those vendors are best-in-class, and are amazing -- it happens! But, some of those vendors definitely are not. Do you think company 3 can convince the awesome vendors to send engineers over to the terrible vendors to fix things up? Yeah, I don't think so. That's not going to happen. It's the ultimate silo, and you're never going to break it.

Would you ever willingly build a company like #2? I hope not. But, if you build a company like #3, you've basically done it anyway: you get all of the walls and none of the opportunities to tear them down later.

There are plenty of other gotchas which come with each of these three scenarios, but I'll leave it there for now. It boils down to this: the proliferation of interconnected vendors leads to a loss of control, and few things are more depressing than that.

Read the whole story
smarkwell
24 days ago
reply
Share this story
Delete

Microsoft open sources SandDance, a visual data exploration tool

3 Shares

SandDance, the beloved data visualization tool from Microsoft Research, has been re-released as an open source project on GitHub. This new version of SandDance has been re-written from the ground up as an embeddable component that works with modern JavaScript toolchains.

SandDance screenshot

The release is comprised of several components that work in native JavaScript or React apps, in addition to using various open source libraries. We hope that with this new architecture, SandDance will be vastly more extensible, enabling new chart types, layers, and interactions, as well as being incorporated into new, vertical applications.

In addition to the raw JavaScript components, SandDance is available as an extension to both Visual Studio Code and Azure Data Studio and has also been re-released as a Power BI Custom Visual. We hope to see other innovative uses of SandDance embedded within many other applications where unit-based data visualization makes sense, such as in machine learning tools. We believe that making it an open source component is the first step towards that goal.

For those unfamiliar with SandDance, it was introduced nearly four years ago as a system for exploring and presenting data using unit visualizations. Instead of aggregating data and showing the resulting sums as bar charts, SandDance shows every single row of a dataset (for datasets up to ~500K rows). It represents each of these rows as a mark that can be colored and organized into different areas on the screen. Thus, bar charts are made of their constituent units, stacked, or sorted.

SandDance screeshots GIF

Different visualizations and layouts are all related to each other so that the units are animated from one position to another. This makes for compelling explanations through following a story using different representations to help show both an insight and how it fits within the context of the rest of the data.

SandDance provides 2D and 3D visualizations, bar charts, scatter plots, stacked charts, density plots, as well as interactive filtering and faceting of the data. Its been showcased in recent PowerBI Summit and CEO Summit keynotes and has been downloaded by more than 300K people to date.

SandDance was created by Microsoft Researchs Visualization and Interactive Data Analysis (VIDA) group, which focuses on a human-centric approach to working with data, exploring areas such as data visualization, immersive analytics, and understanding of machine learning models. You can learn more about their work here.

Learn more about the new SandDance project on GitHub or our companion website. Here, you can try SandDance on any local dataset (nothing gets uploaded to the server), as well as instructions for building local versions or incorporating it into your own projects.

Other questions or feedback? Let us know in the comments below.

 

The post Microsoft open sources SandDance, a visual data exploration tool appeared first on Open Source Blog.

Read the whole story
diannemharris
21 days ago
reply
smarkwell
25 days ago
reply
Share this story
Delete

Name Shadowing Should Be An Operator

1 Share

I recently discovered that in Rust, this is a relatively common operation:

let foo = String::from("foo");
// stuff that needs ownership
let foo = &foo;  
Or this:
let mut vec = Vec::new();
vec.push("a");
vec.push("b");
let vec = vec; /* vec is immutable now */  
This is a particularly permissive form of name-shadowing, which allows you to re-declare a variable in an inner scope that shadows the name of a variable in an outer scope, making the outer variable inaccessible. Almost every single programming language in common use allows you to do this in some form or another. Rust goes a step further and lets you re-declare a variable inside the same scope as another variable.

This, to me, is pretty terrifying, because name-shadowing itself is often a dangerous operation. 90% of the time, name-shadowing causes problems with either temporary or index variables, such as i. These are all cases where almost you never want to name-shadow anything and doing so is probably a mistake.

for(int i = 0; i < width; ++i)
{
  for(int j = 0; j < height; ++j)
  {
    //
    // lots of unrelated code
    //
    
    float things[4] = {1,2,3,4};
    for(int i = 0; i < 4; ++i)
    	box[i][j] += things[i] // oops!
    //      ^ that is the wrong variable!
  }
}
These errors almost always crop up in complex for loop scenarios, which can obviously be avoided with iterators, but this isn’t always an option, espiecally in a lower-level systems programming language like Rust. Even newer languages like Go make it alarmingly easy to name-shadow, although they make it a lot harder to accidentally shoot yourself because unused variables are a compiler error:
foo, err := func()
if err != nil {
	err := bar(foo) // error: err is not used
}
println(err)
Unfortunately, Rust doesn’t have this error, only an unused-variables linter warning. If you want, you can add #![deny(clippy::shadow_unrelated)] to be warned about name-shadowing. However, a lot of Rust idioms depend on the ability to name-shadow, because Rust does a lot of type-reassignment, where the contents of a variable don’t change, but in a particular scope, the known type of the variable has changed to something more specific.
let foo = Some(5);
match foo {
    Some(foo) => println!("{}", foo),
    None => {},
}
Normally, in C++, I avoid name-shadowing at all costs, but this is partially because C++ shadowing rules are unpredictable or counter-intuitive. Rust, however, seems to have legitimate use cases for name-shadowing, which would be reasonable if name-shadowing could be made more explicit.

I doubt Rust would want to change it’s syntax, but if I were to work on a new language, I would define an explicit name-shadowing operator, either as a keyword or as an operator. This operator would redefine a variable as a new type and throw a compiler error if there is no existing variable to redefine. Attempting to redefine a variable without this operator would also throw a compiler error, so you’d have something like:

let foo = String::from("foo");
// stuff that needs ownership
foo := &foo;
Or alternatively:
let mut vec = Vec::new();
vec.push("a");
vec.push("b");
shadow vec = vec; /* vec is immutable now */
While the := operator is cleaner, the shadow keyword would be easier to embed in other constructions:
let foo = Some(5);
match foo {
    Some(shadow foo) => println!("{}", foo),
    None => {},
}
Which method would depend on the idiomatic constructions in the language syntax, but by making name-shadowing an explicit, rather than an implicit action, this allows you to get the benefits of name-shadowing while eliminating most of the dangerous situations it can create.

Unfortunately, most modern language design seems hostile to any feature that even slightly inconveniences a developer for the sake of code safety and reliability. Perhaps a new language in the future will take these lessons to heart, but in the meantime, people will continue complaining about unstable software, at least until we put the “engineer” back in “software engineering”.

Read the whole story
smarkwell
29 days ago
reply
Share this story
Delete

The next chapter of Flyway

1 Share

In early 2010 I was doing some work around deployment automation. As part of this effort, I decided to tackle the problem of rolling out database changes along with the application consistently across environments, I wanted a tool that could be tightly integrated with application deployment, worked with plain SQL, and was simple to use. I was very surprised that my search did not come up with any solution that met my needs. This was very early days for continuous delivery and the tooling landscape wasn’t nearly as mature as it is today. So I decided to roll up my sleeves and build my own. On the 20th of April 2010, Flyway 0.0.1 was released on Google Code (yes, that used to be a thing!) under the Apache 2.0 license.

Not in my wildest dreams could I have imagined that over the 9 years that followed it would grow as incredibly as it did. Flyway is now downloaded millions of times each month. It has established itself as the industry defacto standard for database migrations with plain SQL. It now supports 18 different types of databases. It is available for Java, Android, Windows, macOS, Linux and Docker. There are integrations available for Maven, Gradle, ANT, SBT, Spring Boot, Play, Jenkins, XL Deploy, NPM, and much more. The community has written hundreds of blog articles about it in dozens of languages. People have been doing conference talks about it on every continent.

This vast and ongoing growth has enabled us to launch both a Pro and an Enterprise Edition in December 2017, effectively securing the long term economic viability of the project.

I am very proud of what we have accomplished so far and I am truly humbled by how positively all our hard work has been received.

Flyway and Redgate

Today I am honored to announce that I am passing the baton to a dynamic company with a solid track record in the database tooling space: Redgate Software. Please join me in welcoming them as new stewards of Flyway. They will be combining their vast industry expertise with everything you know and love about Flyway to take the project forward and make it even better than it is today.

As far as I am concerned, I’ll be around to support the transition but then gradually stepping down and taking some long overdue rest. It has been both a pleasure and honor to work with you all on Flyway through the years. I ride deeply satisfied into the sunset, knowing that Flyway’s future has never looked brighter.

From the deepest of my heart, thank you.

Axel Fontaine

Read the whole story
smarkwell
111 days ago
reply
Share this story
Delete

Your nines are not my nines

2 Shares

I've had some occasions of late to peer through the looking glass into a world that I hadn't seen much of previously. Specifically, I'm talking about the world of so-called "cloud" stuff, where you basically pay someone else to build and run stuff for you, instead of doing it yourself.

I'll skip the analysis of build vs. buy and just jump straight to the point where you've chosen "buy". Then you've had a whole bunch of fun outages caused by something going wrong with their services. Finally, you reach the point of a sit-down talk with the vendor to figure things out. Maybe they send some sales people too, or perhaps it's just engineers. You talk for a while, and before long, you realize what happened.

They are huge. They are like a giant which lumbers around while you are a gnat. You are nothing to them.

This becomes obvious when talking about some problem you experienced at the hands of their system. The whole time, their dashboard stayed green because from their point of view, they had tremendous availability. We're talking 99.999% here! Totally legit!

Meanwhile, you were having a really bad day. Nothing was working. Your business was in shambles. Your customers were at your throat yelling for action, and all you could do is point at the vendor. What happened?

Well, this is the point where you find out that their "99.999%" availablity is for their entire system. They see that, and they're good. It's not a problem! Everything is fine.

This also completely misses the fact that for you, everything was failing. It doesn't matter though, since your worst day still won't move the needle on their fail-o-meter. They won't see you. They won't have any idea anything even happened until you complain weeks later.

You are the bug on the windscreen of the locomotive. The train has no idea you were ever there.

The problem is that they weren't monitoring from the customer's perspective. Had they done that, it would have been clear that oodles of requests from some subset of customers were failing. They would have also realized that certain customers had all of their requests failing.

For those customers, there were no nines to be had that day.

Seriously, if you have a multi-tenant system, you owe it to your customers to monitor it from their point of view. Otherwise, how can you possibly know when you've done something that'll leave them in the cold?

Read the whole story
denubis
116 days ago
reply
Sydney, Australia
smarkwell
119 days ago
reply
Share this story
Delete
Next Page of Stories