We interview Mike Hamilton, CTO of Griid about bitcoin mining. Mike is responsible for Griid’s technical and operational success. Previously, he was CEO of a cloud-based cybersecurity platform.
Transcription
Scott: So today we’re talking about Bitcoin mining and specifically we have the CTO of GRIID. And Mike will tell us a little bit about that. So yeah, Mike, what’s with GRIID? I don’t know much about them yet.
Mike: Yeah. So we’re relatively new, but basically we’re mining Bitcoin. Got multi-megawatts worth of power and just mining away. So that’s pretty much it. Wholly- we own all of our own gear and yeah. Just having some fun while we’re doing it.
Scott: Multi-megawatts. That sounds nice.
Mike: It’s had a… It’s been an interesting journey.
Ethan: It sounds like… What are you mining with? What kind of miners?
Mike: Pretty much all S9s right now. Pretty much all BTC, staying with the godfather and keeping it real on the Bitcoin side.
Ethan: Well you need megawatts to make a profit with S9s now.
Mike: Yeah, there’s lots of fun… I was watching the second episode with JL and, you know, some interesting things you can do there with power and all sorts of knobs that you can play with.
Ethan: [crosstalk 00:01:41] What kind of a role- [crosstalk 00:01:43]
Ethan: What kind of pool are you pointed to?
Mike: So we currently use F2Pool and I’ve been super happy with them.
Scott: [crosstalk 00:01:53] Yeah. We’ve been seeing some people switching over to different pools lately. You know, some of them don’t like the big pools, they don’t like supporting certain basic money manufacturers or whatnot.
Mike: Right.
Ethan: Yeah. I’d like to just take this moment here to thank our sponsor for this podcast, NovaBlock mining pool. They’re a new pool and they’re based here in the United States, in North America, and they launched back in August, 2019. And within four short months they’ve managed to become one of the top 15 in the world in terms of public mining pools. And their goal is to decentralize the hash rate with a focus on North America, as China is shifting away mining from China and moving into the United States in North America. They want to be a leader in the mining industry and, you know, change the way that pools are utilized through transparency and educating their customers for the best type of pool for their needs.
Scott: Yeah, so we have an exclusive offer for our listeners. Anyone that wants to test out their mining pool, you can use the invitation code OFFORD18. So that’s O.F.F.O.R.D. one-eight, and you can receive a permanent reduction in your pool fees down to 1.8% fee. So that code is only available until around January 20th, but if you sign up and use that before then, you can have a permanent fee of 1.8%. So if you have an interest in this, you can go to novablock.com and talk to Vincent. His email as [email protected], and you can check out to see if you want to qualify for that, or if you’d like to learn more about mining pools.
Ethan: Check it out guys. So Mike, I’ve got a question for you, and you’ll have to forgive me if it sounds ignorant or not. You’re chief technical officer, correct?
Mike: Correct.
Ethan: And in another life I was actually a chief information officer of a couple of companies, actually. And I know exactly what that job entails. Can you tell me, from your perspective, what’s the difference between a chief information officer and a chief technical officer?
Mike: Yeah. I think, I think for me the biggest difference- and I think it really sort of depends on the type of company that you’re working for. In our case, you know, sort of the information aspect sort of rolls up with me, because we’re obviously fairly small. It’s not like we’ve got hundreds of employees or anything like that. But really what I focus on is how can- what technology is out there, either in open source form or commercial form, or what needs- or what’s missing altogether. And how do we take those, put them together… If we need to go hire developers and build stuff. How do we take the data that we collect and actually make use of it? It’s sort of- anything that falls into the technical bucket sort of all rolled up into one. You know, development, product management… You know, all those sorts of things.
Scott: So who’s the CEO?
Mike: The CEO is a former coworker and obviously now current coworker. Trey Kelly is our CEO.
Scott: Trey. Okay. Okay, cool. I’ve seen him [crosstalk 00:05:20] online and-
Mike: Yeah, and he’s actually based out of Cincinnati.
Ethan: That’s where I am.
Mike: Yeah. That’s where he lives.
Ethan: All right. Well I’ll have to see if I can reach out to him, then, and do a face to face with him.
Mike: Yeah, absolutely.
Ethan: [crosstalk 00:05:33] I promise the Beard’s much more impressive, face to face, then it is… The camera doesn’t catch it.
Mike: Good.
Ethan: Yeah.
Scott: Now what geographical locations are you guys mining in?
Mike: Currently all East coast, but we’re- we’ve got leads on next locations all- sort of all over the place. But all U.S., no interest in going overseas.
Scott: Now why is that? Why do you prefer the U.S.?
Mike: There’s regulatory risk, you know, and keeping it in the USA. Mostly it’s for- I’ve got four kids. Well, most of us have either younger kids or otherwise, and you don’t want to be hauling overseas all the time. It’s easier to have a short flight to get where you need to be than the long flights.
Scott: Absolutely. Family first.
Mike: Family first.
Ethan: Last episode, we were talking with JL about his struggles to get really good power costs. What input can you provide about your power costs? Do you use a power broker, or do you have another system or another way of getting good power costs?
Mike: Good old fashioned hard work hitting the phones and relationships. The power broker is certainly- that was an interesting thread of conversation, and certainly as we look in other areas where we might not have the same set of relationships we have in our current zone, the power broker is definitely an interesting option. The other thing you guys were talking about with the sort of the interruptible sort of time of service. That’s certainly… You can find power companies and their published rate schedules online for the sort of interruptible set up. Even those are sort of in the ballpark of a profitable… But yeah, there’s definitely technical issues in the time of use, interruptible schedule and it turns out going from zero to 10 megawatts in a matter of 10 minutes is not a terribly trivial problem. Actually, going from five or 10 to zero is easy. Effectively going from zero to 10, and what the ramps look like, and how you do that and manage that, is a whole nother set of issues.
Scott: Yeah.
Ethan: Yeah. So what kind of power costs do you and your organization try to strive for? You know, I’m just trying to get a feel for who’s getting the best deals out there.
Mike: Yeah. And I think if you can- I think the easiest way to see that is if you look at the hosting deals that get offered, it’s like most of those are right around, between five and a half, six and a half cents from a hosting perspective. Which means they’re getting it somewhere in the three cent range. If they’re going to… A little markup. Obviously there’s a lot of labor, and moving systems, and fixing hardware onsite from a hosting perspective. So between three and four cents is… If you’re not there, then obviously you have a different set of issues, and trying to reduce your Watts per terahash and trying to squeeze the most you can. And sort of just [inaudible 00:08:50] along.
Mike: Yeah, so we, we, we try to aim beneath what it costs to host. And we don’t do hosting. That’s not our model. We’re mining for ourselves and for our people, it’s not a hosted model. But it’s a nice indicator to have the hosting model out there to sort of understand what the baselines are, and where the facilities are. Because you’re probably not going to host in an area where you’re getting six cent power. So that’s obviously a good indicator is where the hosting sites are. You might be able to squeeze your own deals out, let the other folks do the work.
Scott: And last pick, mine is at 50,000 or something? Yeah, I’m not going to be mining from my own home right now.
Mike: Yeah, exactly. I’ve done that. I’m sure everyone’s done that. And the wife got tired of a) the heat, the noise, and I actually had converted our half completed wine cellar project ended up turning into the data center. And then ultimately gave it up and converted it back to the wine cellar. That’s how much I love mining, that I actually like it more than wine. [crosstalk 00:10:00] That’s a high bar.
Scott: You like crypto more than mine- more than wine. That’s awesome.
Mike: Unfortunately, my wife doesn’t like crypto more than wine. So- (laughs)
Scott: (laughs) Yeah, when I was mining in my basement, I had maybe five S9s down there, and that’s where my wife runs on the treadmill. And man, she was- yeah, it was hot. She didn’t like that very much at all. So I had to get those out of there.
Mike: Yeah, luckily our old house had a sort of an unfinished area, sort of underneath the house that wasn’t really usable, so it’s able to toss them in there. But of course then it’s a little dirty, and I was- I had three 13 GPU rigs in there, so 39 GPUs in there. And they burned some heat.
Scott: (laughs) Yeah, I can imagine.
Ethan: So tell us… I think offline you were telling us that you guys run your miners in containers?
Mike: Yes, we do run in containers.
Ethan: What advantage does that give you?
Mike: I think the biggest thing is ultimately if you look at mining in the broader sense, ultimately you’ve got a couple of different things you’re trying to cover as you’re mining. Obviously you’ve got the power, you’re trying to make sure you can pay for. And then you’ve got the overhead of what it takes to buy the hardware. And then of course the overhead of what it takes to get set up: your PDUs, your network, et cetera. You know, and frankly, people can say good things or bad things, whatever they like about Bitmain… S9s are like tanks. They run in all kinds of conditions, humidity, not humidity, and so ultimately you’re looking at large scale multi megawatt projects. Containers is the easiest and fastest way to get up and running with as little overhead that where your deployed capital is going to money assets. The miners, essentially. So that’s sort of the logic behind that model.
Scott: Now do you manufacture your own containers or do you buy them from a provider?
Mike: We do not currently manufacture our own containers, although that’s one of my fun projects is looking at- there’s obviously lots of really interesting manufacturers of containers out there. I’ve sort of personally looked at four or five, six different versions. The ones we’re currently mining out are fairly basic. But yeah, it’s something we’re looking at on the right way to do that. Whether or not it’s Ambien or if it’s, if you’ve got exhaust fans, or if you’re using mini fans, or exit the existing minor fans to sort of passively with the hot cold separation… All sorts of really interesting problems to solve in the container mining aspect.
Scott: Yeah. Now is it something that you would be willing to share who your current provider is for containers? Or…
Mike: Yeah, I don’t think it’s any super secret, but- we actually have been using the amp boxes. [crosstalk 00:13:07] The BIM and amp boxes. And interesting issues, they come with sort of your Chinese PDUs, and so there’s a little extra work to go back and get the UL rated PDUs dropped in there. But frankly, it’s a fairly simple, and sort of- it’s actually a fairly elegant design really in sort of the thought of detail. I’ve been actually pretty impressed with the quality of the container, the louvers, the [inaudible 00:13:39] blankets, all the things that come along with them I’ve actually been pretty impressed with.
Scott: Yeah, Ethan had actually saw some of those in operations [crosstalk 00:13:46] in Texas. We visited the Bitmain facility down there.
Mike: Oh yeah, down in Rockdale? Yup. Yup, Rockdale.
Scott: [crosstalk 00:13:52] And we saw they had fuses instead of breakers in their PDU. Is that what you’re dealing with?
Mike: Yeah. So like I said, we’re not using the stock PDUs that came, but we are using very similar PDUs that actually do have fuses, as opposed to breakers.
Scott: Okay.
Mike: And then incidentally, the fuses are actually super expensive. If you go down to the Home Depot, they’re six, eight bucks a piece.
Ethan: Oh really? I thought they would be much cheaper.
Mike: Well, I’m- unfortunately, I’m an Alibaba expert, so I was able to get them for about 18 cents a piece. So frankly I’m in the wrong business. I should just be selling $8 fuses on a 18 cents basis.
Ethan: (laughs) Right.
Mike: There’s a little better margins apparently in 20 amp fuses than there is in Bitcoin mining.
Ethan: How frequently do the fuses blow out? Is it a common occurrence? I mean, you said you bought in bulk from Alibaba, so-
Mike: [crosstalk 00:14:53] Yeah, but I think I bought 200, about 250 of… I think maybe to a couple hundred of them. And I think maybe we’re- maybe we’ve used 30 or 40 total. And almost all the 30 or 40 that we burned were either during a bring up of sort of a go live in a new section. So it was more about underlying issues. When we’re in normal operations, they’re almost never burned out.
Ethan: Now you have a background in cybersecurity, is that correct?
Mike: That’s correct.
Ethan: So can you tell Scott and I and the audience some of the security issues that you face in setting up and operating a mine?
Mike: Yeah. Great, great question. There’s been some interesting ones. Coming out of the sort of enterprise, I mean, we had fortune 10, fortune 50 type customers at the last company doing an antivirus software. So while I’m not a security researcher I’ve designed, developed, built and sold security products for, I guess, the better part of the last eight-10 years.
Mike: And I’ve been impressed with the creativity of the… So our first batch of machines we bought were actually used machines. And as I’ve- going through, and getting them spun up and turned on, configured, found… Let’s see how many, maybe a half a dozen or so compromised machines.
Ethan: Oh really?
Mike: So I was able to pull them off the network, and actually go do some forensic analysis on them. And you know, there’s some interesting things where they’ll creatively, the attackers have actually modified the CGI scripts that will actually, if you go in and you’ll set your pool settings, it’ll take the setting, you go back to the page, it’ll show the setting as you set it up. But underneath it’s actually, it’s actually running some nice hash. They’re- they tend to use nice hash typically, the attackers. And then even if you- the only way to really tell is to actually go and actually hit the CG miner API directly. Because they even modify the miner status page to show whatever you configured, even though the CG minor API won’t lie to you, but… It’s pretty fascinating.
Scott: [crosstalk 00:17:22] So that’s how you found-
Ethan: [crosstalk 00:17:24] I think that happened before. One of mine bought a big batch of miners and yeah, they had them running for a little while. They put in their pool setting and everything, and they couldn’t figure out what was going on. And finally they realized, yeah. You know, they did some really tricky things to make it look like it had actually received the correct pool setting, but it didn’t.
Mike: Yeah, so the- it’s only the way, sort of coming from a security background, don’t trust anything. Especially when you’re loading up several thousand used miners that are in who knows what sort of shape. Yeahs so I basically run continuous security audits internally comparing both what the UI says against the CG miner API, and even cross correlating it with both the pool settings and our firmware settings. So it’s interesting, and it was sort of- finding it for the first time was pretty exciting, and sort of gie them a chance to pull it apart. And there’s some creative attackers out there.
Scott: Yeah. Now that sounds like an interesting thing that, might be useful for other people too. I don’t know, is this something that you just custom coded yourself to, to watch for? Or-
Mike: Yeah, it is, but that’s a good point. That’s probably something I should- because we use, obviously we use GitHub for all the fun things we build internally. And maybe I’ll see if I can’t get that published. It’s actually pretty simple, it’s not super difficult. But yeah, I’ll see if maybe I can’t publish some of those sorts of tools and bring some of the- I’ll be the miner security guy, and publish some stuff to help people out.
Scott: [crosstalk 00:19:08] Can you give any tips to our audience out there? Let’s say they have compromised minors. So what’s the course of action? Do they just reload in the firmware, or do they have to do something more extreme? What’s the way to fix it?
Mike: Oh, in this case there was a couple of things. One, what they did is they asked- they incidentally also modified the upgrade CGIs. So they wouldn’t- so you couldn’t just upload the software directly from the UI. So you actually had to install a command line and or do the SD flash. Once you SD flash or command line a flash, then you were in good shape. But they literally modified just about anything that you could modify to keep people from doing-
Scott: [crosstalk 00:19:59] A lot of back doors.
Ethan: A lot of back doors. Wow.
Scott: So your recommendation is to use the SD card flash method if you think or suspect that you have a compromised miner?
Mike: Yeah. And actually, incidentally, call this one of the learning mistakes, is that what we should have done is we should done sort of- at least on the Bitmain side, the IP report reset as we were plugging in. But frankly, we just got so excited when a couple thousand S9s show up. We were just stuffing them in there and powering them on. It’s like, “Oh, we’ll figure it out later.” And we probably should’ve been a little bit more systemic, or systematic rather, in that process and sort of done the IP reset upfront. But certainly the SD flashes, nice safe method, though it is time consuming to flip around jumpers. Although incidentally, we also had some of ours that the- and even I’m not entirely sure exactly how the IP reset, the IP report reset works. But even some of those, when IP report resetting them, came back in a slightly what seemed to be a non-factory state. So either command line software updates or the SD flashes. It’s certainly the safest method though. It is certainly more time consuming.
Scott: Right.
Ethan: Any idea how the compromise happened? I mean obviously, you got them already compromised, but do you have any suspicion, or were they on a vulnerable network? Or yeah, any ideas or thoughts about that?
Mike: Yeah, that’s a good question. It really could be anything. Most likely if you think about the type of people who are running mines, and the type of people who are plugging their computers in to the mine to go update systems or sort of maintain them, probably have hardware wallets, or rather software wallets. They might have- I’ve got… Raven was one of the interesting coins that I played around with and still sort of keep the wallet for. I imagine some of the folks that are operating the minds have similar software maybe for other… So my gut suspicion is that it’s more likely someone came in to do some maintenance on the systems with the compromised system, and there was more sort of a worm effect as opposed to- it’s fairly hard to compromise the system. I mean, if you think about what [inaudible 00:22:25] are doing, they go get some work every 10 minutes and then just hang out for 10 minutes. There’s not a whole lot of opportunity to just sort of bust through and get to the miners directly. It’s more likely just sort of coming in from someone plugging in from the outside. Sort of my theory on it. But just a theory.
Ethan: Sure. It’s always good to know, right?
Mike: Absolutely.
Scott: So Mike, yeah. I was going to ask you maybe some other interesting project that you’ve worked on in the past.
Mike: Yeah, I think one of my favorite projects was I had done some traveling with my wife, getting away from the kids for a little bit, and ended up in fairly remote set of islands, and had sort of a fresh food problem, and sort of came around to growing vegetables inside shipping containers with a little UV light, a couple of Raspberry Pis, and some pumps and actuators, and essentially built a little indoor hydroponic farming set up. And then ultimately got sidetracked by normal life. But as I was saying before, before we hopped on, even some of the same… Using some Raspberry Pis and relays, a lot of the same sort of things we use to control some of the containers and some of the environmental factors and what we deal with. So always a fun project.
Scott: Yeah. So vegetables, that’s what they’re calling it these days, huh?
Ethan: (laughs)
Mike: Yeah, it was actual lettuce. Not [crosstalk 00:24:05] the other kind of lettuce. I’m in Texas, we’re not quite so forward thinking as the Colorado and the Washington and the Californians.
Scott: A different type of leaf that you’re trying to grow? (laughs)
Mike: Yeah, no it was, it was actual lettuce.
Scott: Okay, so you went from using shipping containers to now again using shipping containers?
Mike: [crosstalk 00:24:27] Turns out giant metal boxes have a whole lot more use than just shipping things. Who would have thought?
Scott: Yeah.
Ethan: (laughs)
Mike: It’s like Christmas, where kids open up the Christmas presents and they toss the leg Lego’s aside and they go take the cardboard boxes and go play with those for two days? Turns out that never goes away from us. We grow into adults and we use shipping containers the same way they play with the cardboard boxes.
Scott: And I know my brother wanted to build a house out of shipping containers in the past. So it’s-
Mike: Here in Austin, there’s a shipping container bar. It’s super cool, they took a bunch of shipping containers and welded them together in various shapes and turned it into a bar.
Scott: Awesome.
Ethan: That sounds pretty cool. Yeah, I’d like to definitely go there.
Mike: Yeah, you should come check it out. Austin’s a good town.
Scott: Mike, can you tell us a little bit more about what got you into mining? How did you get started? Did you just wake up one day and said “I’m done with lettuce, I’m going to do something else?” Or what was your segue in?
Mike: Yeah, so sort of coming out of school, coming out of college I was actually an ASIG designer, so I actually designed ASIGs for half a dozen years. Wanted to be a designer for GPUs, but that market consolidated right as I was coming out of school. The 3D effects and all those guys all sort of got merged into the current [inaudible 00:25:57] set. So I ended up doing network processor development. So had it sort of embedded basic chip design experience. Ended up in security, and then sort of last couple of companies had a coworker who was into it, incidentally Trey CEO got me into it. He just thought that I was, I’d be interested in it just because of the- really is more so around blockchain technology and what the implications are for that. And so he introduced me to it about three years ago, and that’s turned into the 39 GPUs in my basement and unhappy wife and-
Scott: Wine? crosstalk 00:26:37 Wine shortages?
Mike: And a smaller wine supply. And then it was time to move on from the corporate side and got together with Trey and got the site down. And that’s what we’re doing.
Scott: A lot of people in this industry, when they hear the term ASIG: application specific integrated circuit, they immediately just think Bitcoin miners. What a lot of people don’t know is application specific integrated circuits had been used for a lot of other things. Can you explain to our audience what other things that ASIGs are used for?
Mike: Yeah, absolutely. They’re used for all sorts of stuff. And frankly, the GPU is in fact an ASIG, it’s just an ASIG for GPU… For sort of the repeatable, mathematical high memory intensive issues. But I mean really ASIGs, Raspberry Pi. The little single board computers. That’s an ASIG that just happens to be just a… It’s got the network onboard, regular CPU on board. Like I said, my background came out of the network processor side, so we built network processors that were used in all the eNodeBs for cell towers. So you know, for processing traffic, classifying network traffic, things like that. I mean, just about any device your cell phone, tons of ASIGs in there. It’s an interesting space. Been out of it for a while now, but still try to follow as close as I can.
Scott: What would you say is the major difference between the ASIGs used for mining Bitcoin and just the every day ASIGs that are in our lives? What is the main difference between those two types?
Mike: I think from the Bitcoin ASIGs isle is, I think really how simple they are. I mean obviously there’s- it’s never simple to build an ASIG and… You’ve got the silicon masks and the metal masks and all of those things. But ultimately when you look at- shout out to [inaudible 00:28:50] chip, you basically have one little block that gets replicated a million times. You put a little control logic around it, there’s still serial ports running on these things, and that’s how they talk. A little I-squared-C, I-squared-C has been around for decades. It’s really more about the ASIG technology and the difficulty in the manufacturing side. The chips themselves are- it’s basically an arm- some arm IP, which ultimately you can… An ARM’s done an unbelievable job in that model of creating the sort of embedded chip or the embedded controller IP and given the tools to wrap around it.
Mike: That by itself has radically increased the speed at which companies like Bitmain and Samsung can get these chips out the door. My first ASIG company, we actually had our own three or four different microcontroller microprocessor designs all in house. That was really before ARM had become what ARM is today. And certainly I can imagine how much quicker we could have done that. [crosstalk 00:29:56] Talking about those early designs with the off the shelf, two gigahertz, quad-core chip. Where the IP is all done for you, lay out’s all done. It’s just literally drop it in and you know, put your supporting stuff around it.
Scott: So there’s some people that think that maybe Bitmain might come out with the five nanometer miner in 2020. Personally, I don’t think it might happen until maybe 2022. What are your thoughts on five nanometer for mining?
Mike: Yeah, I mean for context purposes, the last chip that I worked on before I got out of the ASIG space was 90 nanometer and I- we were getting ready to work on a 65 manometer chip. And that was cutting edge at the time, obviously. The interesting thing-
Scott: We’re now down to a 10th of that. So-
Mike: Yeah. And actually the first one, the first chip I worked on I think was a 0.2 0.33 micron. So 230 nanometer, I think it was the first one, first chip that I worked on. So obviously leaps and bounds in the last 15, 20 years. But in terms of the five nanometer, one of the interesting things about the actual circuits is ultimately you’re getting down to single layers of atoms as the gates on these transistors. And ultimately there’s only so much further you can go.
Scott: Yeah.
Mike: And when you think about it, I mean obviously there’s thousands and millions of these ASIG chips coming out. But ultimately you’re competing against, for capacity in TSMC or Samsung or you know, whichever fab of choice for CPUs or for phones. So there’s a capacity issue, and then there’s just a pure technology of- especially in a power hungry and a power sensitive environment, you start to get to single double layers of atoms, and the amount of leakage and just raw power that is just thrown away because of the thickness of the actual gates. So, like I said, I’m a little bit out of touch in terms of keeping up with the sort of the manufacturing technology, but ultimately running into physics problems. And there’s certainly lots of research out there on carbon nano tubes, and other sorts of ways to essentially create or replicate transistor style behavior with even smaller form factors and less leakage, faster switching, et cetera.
Scott: Yeah. Well, thanks for your time. We’re going to cut it off here soon, but is there anything else that you want to tell us about GRIID, or what you guys are up to? Or where we could find you?
Mike: Yeah, so I’m on Twitter. I’m @MikeHamilton, nice and easy. And that’s probably the easiest place to get ahold of me. But yeah, like I said, I’ll be, as your suggestion, and maybe I’ll publish some on my security tools and see if it can help people out and yeah, happy to talk with anyone that’s interested.
Scott: This episode was sponsored by NovaBlock Mining Pool, and you can go to their website when you sign up. Use the invitation code OFFORD18