[{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/amprs/","section":"Tags","summary":"","title":"Amprs","type":"tags"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/categories/architecture/","section":"Categories","summary":"","title":"Architecture","type":"categories"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/architecture/","section":"Tags","summary":"","title":"Architecture","type":"tags"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/design/","section":"Tags","summary":"","title":"Design","type":"tags"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/design-decisions/","section":"Tags","summary":"","title":"Design-Decisions","type":"tags"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/","section":"From homelab to hybrid cloud","summary":"","title":"From homelab to hybrid cloud","type":"page"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/infrastructure/","section":"Tags","summary":"","title":"Infrastructure","type":"tags"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/racr/","section":"Tags","summary":"","title":"Racr","type":"tags"},{"content":"TL;DR: RACR (Requirements, Assumptions, Constraints, Risks) captures what a design must deliver and what it is standing on. AMPRS (Availability, Manageability, Performance, Recoverability, Security) evaluates how each decision affects the five infrastructure qualities. Use RACR to frame the problem, AMPRS to judge the answer, and trace every decision back to both.\nWhile preparing for my VCF Architect certification, I ran into a topic that goes well beyond the exam. RACR and AMPRS are things I already apply every day, often without naming them, whenever I evaluate a design or a proposal. Studying them in a structured way made me stop and reflect, and this post is the result of that reflection.\nMost infrastructure designs don\u0026rsquo;t fail because someone picked the wrong product. They fail because nobody wrote down why a choice was made, what was assumed to be true, and what it would cost elsewhere. Six months later, someone asks \u0026ldquo;why is this cluster stretched?\u0026rdquo; and the only answer is \u0026ldquo;it seemed like a good idea.\u0026rdquo;\nRACR and AMPRS are the two lenses I use to avoid that. They come from the VMware/VCDX design methodology world, but they work for anything: a vSphere cluster, a Kubernetes platform, a backup strategy, or evaluating whether a vendor proposal actually makes sense.\nWhat is RACR? # RACR is a design-framing method that classifies every piece of input into Requirements, Assumptions, Constraints and Risks before any solution is chosen.\nEach category has a precise meaning, and mixing them is the most common mistake:\nItem Definition Example Requirement Something the design must deliver. Functional (what it does) or non-functional (how well). \u0026ldquo;Tier-1 VMs must tolerate the loss of one host with no service impact.\u0026rdquo; Assumption Something believed to be true but not verified. Every assumption needs an owner and a validation step. \u0026ldquo;The customer\u0026rsquo;s WAN link has at least 1 Gbps available for replication.\u0026rdquo; Constraint Something that limits the design choices and cannot be changed by the architect. \u0026ldquo;Must reuse the existing Fibre Channel array until 2028.\u0026rdquo; Risk Something that may happen and harm the design goals, with a mitigation. \u0026ldquo;Single storage array is a single point of failure for all clusters.\u0026rdquo; A useful rule of thumb: if you can negotiate it, it\u0026rsquo;s a requirement; if you can\u0026rsquo;t, it\u0026rsquo;s a constraint; if you haven\u0026rsquo;t checked it, it\u0026rsquo;s an assumption; if it could go wrong, it\u0026rsquo;s a risk.\nWhat is AMPRS? # AMPRS is an evaluation model that scores every design decision against five infrastructure qualities: Availability, Manageability, Performance, Recoverability and Security.\nAvailability — Can the service stay up when components fail? (HA, redundancy, fault domains) Manageability — Can the team operate it day-2? (automation, monitoring, lifecycle, skills) Performance — Does it meet throughput, latency and capacity targets under real load? Recoverability — When something breaks badly, how fast and how completely do we get back? (RPO/RTO, backups, DR) Security — Does it protect confidentiality and integrity? (segmentation, identity, hardening, compliance) The key idea: no decision is neutral. Almost every choice improves one quality and costs another. AMPRS forces you to say that out loud.\nWhy these frameworks matter # Without structure, design reviews turn into debates between preferences. With RACR and AMPRS:\nDecisions become traceable. Each decision cites the requirement it satisfies and the constraint it respects. Hidden assumptions surface early. An unvalidated assumption is a risk waiting to happen, and RACR puts it on the table. Trade-offs are explicit. \u0026ldquo;We gained Performance, lost Manageability, and here\u0026rsquo;s how we compensate\u0026rdquo; is a defensible statement. \u0026ldquo;It\u0026rsquo;s faster\u0026rdquo; is not. Reviews get faster. A reviewer (or a VCDX panel) can follow the logic instead of reverse-engineering it. How RACR and AMPRS work together # Think of it as a pipeline: RACR defines the problem space, design decisions live inside it, and AMPRS evaluates each decision\u0026rsquo;s impact.\nStep 1: RACR frames the problem # Before drawing any diagram, collect inputs from stakeholders and classify them. Number each item (R01, A01, C01, RK01) so it can be referenced later.\nStep 2: Decisions are made against RACR # Every design decision should answer: which requirement does this satisfy, and which constraint shapes it? If a decision doesn\u0026rsquo;t trace to any requirement, question why it exists.\nStep 3: AMPRS scores the impact # For each decision, mark the effect on each quality: positive (+), negative (−) or neutral (0). Negative impacts must either be accepted with justification or mitigated.\nStep 4: The loop closes # Negative AMPRS impacts often create new risks. Those go back into the RACR register. The design is \u0026ldquo;done\u0026rdquo; when every risk has a mitigation or an explicit acceptance.\nA practical example: stretched cluster or not? # Say the input is: \u0026ldquo;the business wants zero data loss if a datacenter fails.\u0026rdquo; Classified with RACR:\nR01: RPO = 0 for Tier-1 workloads. R02: RTO ≤ 15 minutes. C01: Two sites, 40 km apart, existing dark fiber. A01: Round-trip latency between sites is below 5 ms (to be validated with a network test). RK01: A split-brain scenario if the witness is unreachable. Two options, evaluated with AMPRS:\nQuality Option A: vSAN stretched cluster Option B: async replication + SRM Availability + Automatic restart on surviving site 0 Requires failover orchestration Manageability − Witness host, site affinity rules, more complex upgrades + Simpler clusters, runbooks in SRM Performance − Synchronous writes add inter-site latency + Local write latency only Recoverability + RPO = 0, RTO in minutes − RPO \u0026gt; 0 (fails R01) Security 0 Same trust domain across sites 0 Same Option B fails R01, so it\u0026rsquo;s out regardless of its manageability advantage. Option A is chosen, but its negative impacts become new entries: a risk for write latency (mitigated by validating A01) and a manageability risk (mitigated by documenting witness operations). That is the whole method in one table.\nCommon mistakes # Writing solutions as requirements. \u0026ldquo;Must use vSAN\u0026rdquo; is not a requirement. It\u0026rsquo;s either a constraint (if truly imposed) or a decision you haven\u0026rsquo;t justified yet. Leaving assumptions unvalidated. An assumption with no owner and no validation date is just a risk you hid. Treating constraints as optional. If budget or hardware is fixed, it\u0026rsquo;s a constraint. Designing around it as if it weren\u0026rsquo;t wastes everyone\u0026rsquo;s time. Scoring only the positives in AMPRS. A decision with only \u0026ldquo;+\u0026rdquo; marks usually means the analysis wasn\u0026rsquo;t honest. Risks without mitigations. Listing a risk is half the job. State the mitigation or the explicit acceptance, and who accepted it. Skipping Manageability. It\u0026rsquo;s the quality that hurts most on day 2 and the one most often ignored on day 0. FAQ # What does RACR stand for in infrastructure design? Requirements, Assumptions, Constraints and Risks. It\u0026rsquo;s a method for classifying design inputs before choosing a solution, so every later decision can be traced back to what the business needs and what limits the design.\nWhat does AMPRS stand for? Availability, Manageability, Performance, Recoverability and Security. These are the five infrastructure design qualities used to evaluate the positive and negative impact of each design decision, popularized by the VMware VCDX methodology.\nWhat is the difference between a constraint and an assumption? A constraint is a known, fixed limitation the architect cannot change, like existing hardware. An assumption is something believed true but not yet verified, like available bandwidth. Assumptions need validation; constraints need to be respected.\nIs RACR and AMPRS only for VMware designs? No. They originated in VMware design methodology, but they apply to any infrastructure: Kubernetes platforms, cloud landing zones, networks, backup strategies or evaluating a vendor proposal. They are vendor-neutral thinking tools.\nHow do I document design decisions using AMPRS? For each decision, record the justification, the requirements and constraints it relates to, and the impact on each quality as positive, negative or neutral. Every negative impact should reference a mitigation or a risk entry.\nConclusion # RACR tells you what you are solving and what you are standing on. AMPRS tells you what each answer costs. Neither is complicated, and that\u0026rsquo;s the point: the value isn\u0026rsquo;t in the acronyms, it\u0026rsquo;s in the discipline of writing things down before arguing about them. Next time you review a design, including your own, ask two questions for every decision: which requirement drove this? and what did it cost in AMPRS? If there\u0026rsquo;s no answer, you\u0026rsquo;ve found your next piece of work.\nReferences # The Nutanix Design Guide (First Edition) Conceptual Design: RCARs \u0026amp; AMPRS, VCDX Style — Stephan McTighe Journey to VMware Cloud Foundation 9.0 Architect Certification: A Complete Preparation Guide — cnuggets ","date":"24 September 2026","externalUrl":null,"permalink":"/posts/racr-amprs/","section":"Posts","summary":"","title":"RACR and AMPRS: a two-lens framework for infrastructure design decisions","type":"posts"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/risk/","section":"Tags","summary":"","title":"Risk","type":"tags"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/vcdx/","section":"Tags","summary":"","title":"Vcdx","type":"tags"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/virtualization/","section":"Tags","summary":"","title":"Virtualization","type":"tags"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/categories/vmware/","section":"Categories","summary":"","title":"VMware","type":"categories"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/categories/troubleshooting/","section":"Categories","summary":"","title":"Troubleshooting","type":"categories"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/troubleshooting/","section":"Tags","summary":"","title":"Troubleshooting","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/vlsr/","section":"Tags","summary":"","title":"Vlsr","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/vmware/","section":"Tags","summary":"","title":"Vmware","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/vrms/","section":"Tags","summary":"","title":"Vrms","type":"tags"},{"content":"If you manage vSphere Replication through the API and are hitting intermittent Connection refused errors on port 8043, this is likely what\u0026rsquo;s happening. The error looks like this:\nUnable to retrieve pairs from extension server at https://vrms-appliance.example.local:8043. Unable to connect to vSphere Replication Management Server at https://vrms-appliance.example.local:8043. Reason: https://vrms-appliance.example.local:8043 invocation failed with \u0026#34;java.net.ConnectException: Connection refused\u0026#34;, ErrorCode=2.99.2, OpID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx hms.service (the vSphere Replication Management Server Java process) listens on port 8043. When it\u0026rsquo;s not there, API calls fail. The process wasn\u0026rsquo;t missing because of a network problem — it was restarting.\nFollowing the restart counter # The first useful number came from systemctl show:\nroot@vrms-appliance [ ~ ]# systemctl show hms.service --property=NRestarts NRestarts=13 13 restarts since the last VRMS reboot 13 days prior — roughly one crash per day. hbrsrv.service (the replication engine) had been running continuously for those same 13 days, so the appliance itself was stable. hms was the one cycling.\nThe journal had the mechanism:\nApr 14 11:16:28 vrms-appliance systemd[1]: hms.service: Watchdog timeout (limit 5min)! Apr 14 11:16:28 vrms-appliance systemd[1]: hms.service: Killing process 1725075 (java) with signal SIGABRT. Apr 14 11:16:29 vrms-appliance systemd-coredump[2582430]: Process 1725075 (java) of user 980 dumped core. Apr 14 11:16:29 vrms-appliance systemd[1]: hms.service: Main process exited, code=dumped, status=6/ABRT Apr 14 11:16:29 vrms-appliance systemd[1]: hms.service: Failed with result \u0026#39;watchdog\u0026#39;. Apr 14 11:16:34 vrms-appliance systemd[1]: hms.service: Scheduled restart job, restart counter is at 13. systemd\u0026rsquo;s watchdog kills the process if it stops responding to health checks within the configured interval (5 minutes here) and restarts it. hms was consistently missing that window.\nToo many open files # The kernel audit log pointed at the java process:\nApr 14 11:16:28 vrms-appliance kernel: audit: type=1701 audit(1776176188.958:11942): auid=4294967295 uid=980 gid=980 ses=4294967295 subj=unconfined pid=1725075 comm=\u0026#34;java\u0026#34; exe=\u0026#34;/usr/java/jre-vmware/bin/java\u0026#34; sig=6 res=1 And hms-stderr.log had the actual error, repeating on a roughly 2-second interval for several minutes before the crash:\nApr 14, 2026 11:16:08 AM org.apache.tomcat.util.net.Acceptor run SEVERE: Socket accept failed java.io.IOException: Too many open files at java.base/sun.nio.ch.Net.accept(Native Method) at java.base/sun.nio.ch.ServerSocketChannelImpl.implAccept(Unknown Source) at java.base/sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source) at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:449) at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:129) The Tomcat acceptor thread was trying to accept new connections and hitting the file descriptor limit. Once you can\u0026rsquo;t accept connections, the process can\u0026rsquo;t respond to systemd\u0026rsquo;s watchdog health check either — so systemd kills it.\nAbout 2.5 hours after the last restart, the process was already at 14,472 open file descriptors against a limit of 20,000:\nroot@vrms-appliance [ ~ ]# cat /proc/2582480/limits | grep -i \u0026#34;open files\u0026#34; Max open files 20000 20000 files root@vrms-appliance [ ~ ]# ls /proc/2582480/fd | wc -l 14472 72% of the limit gone in under three hours. At that rate, the process hits the ceiling well before the end of the day.\nTracing it to schedule-health-checks # A Broadcom engineer suggested looking at schedule-health-checks in hms-configuration.xml:\nroot@vrms-appliance [ ~ ]# cat /opt/vmware/hms/conf/hms-configuration.xml | grep \u0026#34;\u0026lt;schedule-health-checks\u0026gt;\u0026#34; \u0026lt;schedule-health-checks\u0026gt;true\u0026lt;/schedule-health-checks\u0026gt; schedule-health-checks tells hms to periodically poll replication health on all protected VMs. Each cycle opens connections that are never properly closed — so file descriptors accumulate until the process runs out and crashes.\nHow to check if you\u0026rsquo;re affected # Run these commands on your VRMS appliance via SSH.\n1. Check if hms.service has been restarting:\nsystemctl show hms.service --property=NRestarts Any value above 0 means the service has crashed and been restarted since the last appliance reboot. Cross-reference with uptime:\nuptime If the appliance has been up for weeks but NRestarts is in the double digits, something is cycling hms regularly.\n2. Confirm the restarts are caused by a watchdog timeout:\njournalctl -u hms.service --no-pager | grep -i \u0026#34;watchdog\\|Failed with result\u0026#34; You\u0026rsquo;re looking for lines like:\nhms.service: Watchdog timeout (limit 5min)! hms.service: Failed with result \u0026#39;watchdog\u0026#39;. If you see those, the process is hanging — not crashing for an unrelated reason.\n3. Check current file descriptor usage:\nGet the PID of the running hms process:\nsystemctl show hms.service --property=MainPID The output will look like:\nMainPID=2582480 Use that number in the next two commands — replacing 2582480 with whatever value you got:\ncat /proc/2582480/limits | grep -i \u0026#34;open files\u0026#34; ls /proc/2582480/fd | wc -l The first command shows the configured limit (Max open files). The second shows how many file descriptors the process currently has open. If the count from the second command is already a large fraction of the limit — and the service restarted recently — the leak is active.\n4. Confirm the parameter is enabled:\ncat /opt/vmware/hms/conf/hms-configuration.xml | grep \u0026#34;\u0026lt;schedule-health-checks\u0026gt;\u0026#34; If it returns \u0026lt;schedule-health-checks\u0026gt;true\u0026lt;/schedule-health-checks\u0026gt;, you\u0026rsquo;re hitting this issue.\nFix # Steps:\nTake a snapshot of the appliance first.\nSSH into the appliance and stop hms.service:\nsystemctl stop hms.service Edit the configuration file:\nvi /opt/vmware/hms/conf/hms-configuration.xml Change:\n\u0026lt;schedule-health-checks\u0026gt;true\u0026lt;/schedule-health-checks\u0026gt; To:\n\u0026lt;schedule-health-checks\u0026gt;false\u0026lt;/schedule-health-checks\u0026gt; Wait approximately two minutes for existing open sockets to time out and close.\nStart the service:\nsystemctl start hms.service Confirm NRestarts is now 0:\nsystemctl show hms.service --property=NRestarts You can also verify how long hms has been up without issue:\nsystemctl status hms.service | grep Active After the fix # The following day, NRestarts was back to 0 and has stayed there.\nroot@vrms-appliance [ ~ ]# systemctl show hms.service --property=NRestarts NRestarts=0 The Broadcom engineer on this case was genuinely helpful — they pointed us to the schedule-health-checks parameter and, after we confirmed the fix, took the time to document it officially. That kind of follow-through is worth calling out.\nBroadcom published KB 437468 covering this issue. The KB confirms the root cause: health check connections for Enhanced Replication mappings are never properly closed, so file descriptors accumulate until the process crashes. Affected and fixed versions:\nvSphere Replication Live Site Recovery Affected 9.x with large-scale ESXi hosts using Enhanced Replication 9.0.3, 9.0.4 Fixed 9.0.2.3 9.0.5 Upgrading to the fixed version removes the need for the schedule-health-checks workaround entirely.\n","date":"20 April 2026","externalUrl":null,"permalink":"/posts/vrms-hms-connection-refused/","section":"Posts","summary":"","title":"VRMS Connection refused: tracking down an hms.service file descriptor leak","type":"posts"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/vsphere/","section":"Tags","summary":"","title":"Vsphere","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/vsphere-replication/","section":"Tags","summary":"","title":"Vsphere-Replication","type":"tags"},{"content":"","date":"2 April 2026","externalUrl":null,"permalink":"/tags/benchmark/","section":"Tags","summary":"","title":"Benchmark","type":"tags"},{"content":"","date":"2 April 2026","externalUrl":null,"permalink":"/tags/fio/","section":"Tags","summary":"","title":"Fio","type":"tags"},{"content":"","date":"2 April 2026","externalUrl":null,"permalink":"/tags/lsi/","section":"Tags","summary":"","title":"Lsi","type":"tags"},{"content":"","date":"2 April 2026","externalUrl":null,"permalink":"/tags/performance/","section":"Tags","summary":"","title":"Performance","type":"tags"},{"content":"","date":"2 April 2026","externalUrl":null,"permalink":"/tags/pvscsi/","section":"Tags","summary":"","title":"Pvscsi","type":"tags"},{"content":"The VMware docs say to use PVSCSI for storage-intensive workloads. The internet agrees. And it\u0026rsquo;s not wrong — but after running the same fio workload across five concurrent VMs with both controllers, the controller type turned out to be the less interesting variable. SIOC was doing all the heavy lifting.\nThis is a follow-up to the fio_benchmark post. Same script, same 200 GiB virtual disks, five VMs hitting storage simultaneously across four configurations: LSI without SIOC, LSI with SIOC, PVSCSI without SIOC, and PVSCSI with SIOC.\nThe test setup # Five VMs (gus_fio_1 through gus_fio_5), all on the same vSphere cluster, all running fio_benchmark simultaneously against a dedicated 200 GiB virtual disk. Each test ran for 300 seconds per profile. The target was /dev/sdb — a dedicated disk with no OS or data on it.\nVM configuration: Ubuntu Server 24.04, 2 vCPU, 4 GB RAM. Identical across all five VMs for each test run.\nEach VM ran the following command simultaneously:\nsudo ./fio_benchmark.sh -t /dev/sdb -r 300 -s -m -r 300 sets 300 seconds per test profile, -s runs the iodepth sweep, -m runs the RW-mix sweep.\nFour configurations:\nLSI Logic SAS (LSI) — the default controller vSphere assigns when you click through the VM creation wizard VMware Paravirtual (PVSCSI) — VMware\u0026rsquo;s purpose-built paravirtual SCSI controller Each repeated with SIOC (Storage I/O Control) disabled and enabled WARNING — fio_benchmark writes directly to the block device you specify. Running it against any disk containing data will destroy that data immediately. Test disks only. Without SIOC: the storage lottery # With five VMs running simultaneously and SIOC off, what you get is a race. Whoever submits I/O first gets the queue — and the results reflect exactly that.\nSequential 128K read — LSI, SIOC off # VM Bandwidth Latency (avg) gus_fio_1 401 MiB/s 9.95 ms gus_fio_2 1,148 MiB/s 3.47 ms gus_fio_3 203 MiB/s 19.63 ms gus_fio_4 326 MiB/s 12.22 ms gus_fio_5 204 MiB/s 19.60 ms gus_fio_2 pulled 1,148 MiB/s while gus_fio_3 and gus_fio_5 were stuck at ~204 MiB/s. Five identical VMs, same workload, same datastore — and one of them got 5.6× more bandwidth than another. Not a configuration difference. Just queue timing.\nThe latency spread confirms it: gus_fio_2 at 3.47 ms average, gus_fio_3 at 19.63 ms. The \u0026ldquo;winner\u0026rdquo; VM was completing I/O 5.7× faster because it had the queue.\nPVSCSI has the same problem:\nSequential 128K read — PVSCSI, SIOC off # VM Bandwidth Latency (avg) gus_fio_1 191 MiB/s 20.85 ms gus_fio_2 382 MiB/s 10.44 ms gus_fio_3 648 MiB/s 6.15 ms gus_fio_4 382 MiB/s 10.45 ms gus_fio_5 191 MiB/s 20.85 ms Different winner, same dynamic. gus_fio_3 grabbed 648 MiB/s while gus_fio_1 and gus_fio_5 sat at 191 MiB/s — a 3.4× gap. The controller changed; the unfairness didn\u0026rsquo;t.\nThis is what happens when multiple VMs share storage without any fairness mechanism. It\u0026rsquo;s not predictable, it\u0026rsquo;s not proportional, and it changes every time you run the benchmark.\nThe iodepth sweep reveals another layer # The iodepth sweep (4K randread, single job, stepping from depth 1 to 128) shows something interesting about how the different VMs behave under contention.\ngus_fio_4 iodepth sweep — LSI, SIOC off:\niodepth IOPS Avg Latency 16 32,470 0.49 ms 32 40,120 0.80 ms 64 40,319 1.59 ms 128 40,735 3.14 ms gus_fio_4 could keep scaling well past iodepth 32 and held over 40K IOPS through depth 128. Meanwhile:\ngus_fio_2 iodepth sweep — LSI, SIOC off:\niodepth IOPS Avg Latency 16 22,399 0.71 ms 32 30,885 1.03 ms 64 19,845 3.22 ms 128 19,274 6.64 ms gus_fio_2 peaked at depth 32 (30,885 IOPS) then fell off a cliff at depth 64 — down to 19,845 IOPS. That\u0026rsquo;s the storage queue saturating: when gus_fio_2 pushed too many concurrent I/Os, they backed up and latency climbed enough to reduce effective throughput. The command queue hit its ceiling, and adding more depth made things worse.\nThis is the most useful diagnostic the sweep provides. If your IOPS curve peaks then drops as you increase depth, you\u0026rsquo;ve found the point where your storage (or hypervisor queue) is saturated.\nWhat SIOC actually does # SIOC (Storage I/O Control) adds a congestion management layer at the datastore level. When it detects that latency is climbing above a threshold (configurable, defaults to 30ms for HDD datastores), it starts throttling VMs that are issuing disproportionately large amounts of I/O.\nSequential 128K read — LSI, SIOC on # VM Bandwidth Latency (avg) gus_fio_1 406 MiB/s 9.82 ms gus_fio_2 401 MiB/s 9.95 ms gus_fio_3 292 MiB/s 13.67 ms gus_fio_4 331 MiB/s 12.07 ms gus_fio_5 293 MiB/s 13.64 ms gus_fio_2\u0026rsquo;s 1,148 MiB/s became 401 MiB/s — a 65% reduction. gus_fio_3, which had been starved to 203 MiB/s, came up to 292 MiB/s. The range compressed from 203–1,148 MiB/s down to 292–406 MiB/s.\nSIOC didn\u0026rsquo;t increase total aggregate throughput — the datastore has the same physical capacity either way. What it did was stop one VM from taking 5× the fair share.\ngus_fio_3 and gus_fio_5 still lag behind gus_fio_1 and gus_fio_2 even with SIOC enabled. This is consistent across all test runs and suggests they\u0026rsquo;re on a different physical disk or VMFS extent within the same datastore — a placement issue that SIOC can\u0026rsquo;t fix.\nPVSCSI with SIOC shows the same pattern:\nSequential 128K read — PVSCSI, SIOC on # VM Bandwidth Latency (avg) gus_fio_1 285 MiB/s 14.00 ms gus_fio_2 568 MiB/s 7.02 ms gus_fio_3 576 MiB/s 6.93 ms gus_fio_4 569 MiB/s 7.01 ms gus_fio_5 285 MiB/s 14.02 ms SIOC compressed the PVSCSI variance from a 3.4× spread (191–648 MiB/s) down to 2× (285–576 MiB/s). gus_fio_1 and gus_fio_5 are still consistently half the bandwidth of the other three — the same two VMs that lagged with LSI. The controller changed; the placement didn\u0026rsquo;t.\nThe RW-mix sweep — SIOC\u0026rsquo;s fairness is most visible here # The RW-mix sweep (4K randrw, 8 jobs, iodepth 8, stepping write % from 0% to 100%) shows how IOPS degrade as write pressure increases. The write penalty curve tells you a lot about storage health.\nLSI, SIOC off — gus_fio_4 RW-mix sweep:\nWrite % IOPS Avg Latency 0% (pure read) 40,585 1.56 ms 50% 33,997 1.92 ms 100% (pure write) 29,838 2.13 ms Clean, gradual degradation. gus_fio_4 was the lucky VM in this run.\nLSI, SIOC off — gus_fio_3 RW-mix sweep:\nWrite % IOPS Avg Latency 0% 19,189 3.32 ms 50% 16,043 4.05 ms 100% 14,888 4.29 ms gus_fio_3 started at 19K IOPS where gus_fio_4 started at 40K. Same datastore, same moment in time. Without SIOC, the spread between the \u0026ldquo;winning\u0026rdquo; and \u0026ldquo;losing\u0026rdquo; VM at pure read is more than 2× — and that gap persists across the entire write penalty curve.\nPVSCSI vs LSI: what the numbers actually say # The paravirtual SCSI controller eliminates hardware emulation overhead. Instead of the hypervisor pretending to be an LSI Logic SAS card, PVSCSI uses a shared ring buffer that the guest driver talks to directly. Fewer context switches, more efficient I/O batching, theoretically better throughput.\nIn practice, with these VMs and this datastore, the difference was workload-specific.\nThroughput and latency (gus_fio_1 only, SIOC off) # Test Controller IOPS Bandwidth Avg Latency 4K rand read LSI 19,238 75.2 MiB/s 6.63 ms 4K rand read PVSCSI 20,193 78.9 MiB/s 6.32 ms 4K rand write LSI 17,357 67.8 MiB/s 3.67 ms 4K rand write PVSCSI 15,451 60.4 MiB/s 4.13 ms Random write LSI 5,146 321.6 MiB/s 12.37 ms Random write PVSCSI 1,653 103.3 MiB/s 38.69 ms Seq write LSI 8,960 280.0 MiB/s 1.76 ms Seq write PVSCSI 5,450 170.3 MiB/s 2.92 ms Random reads are essentially a wash. The write picture is harder to read cleanly: 4K random writes are close, but the larger-block random write (64K bs, 8 jobs) shows PVSCSI at 3× fewer IOPS and 3× higher latency than LSI. Sequential writes follow the same direction — PVSCSI about 40% lower bandwidth.\nThese tests ran simultaneously across all five VMs, so the results aren\u0026rsquo;t just a controller comparison — they\u0026rsquo;re a controller-under-shared-contention comparison. The I/O batching behavior of PVSCSI\u0026rsquo;s ring buffer can shift contention dynamics when multiple VMs hit the same datastore concurrently.\nCPU overhead: where PVSCSI actually delivers # The throughput picture above is mixed. The CPU picture is not.\nTest Controller CPU (usr / sys) IOPS/CPU% Seq read LSI 0.68% / 2.09% 8,563 Seq read PVSCSI 0.48% / 1.51% 9,851 Seq write LSI 2.85% / 2.10% 1,807 Seq write PVSCSI 1.01% / 1.00% 2,701 4K rand read LSI 0.26% / 0.77% 18,640 4K rand read PVSCSI 0.20% / 0.63% 24,260 4K rand write LSI 0.75% / 1.40% 8,104 4K rand write PVSCSI 0.44% / 1.01% 10,666 PVSCSI consistently burns less CPU per IOPS across every test:\nSequential write: LSI uses 4.95% total CPU for 8,960 IOPS. PVSCSI uses 2.01% total CPU for 5,450 IOPS. The bandwidth regression is real, but so is the 59% CPU reduction. 4K random read: PVSCSI is 30% more efficient (24,260 vs 18,640 IOPS/CPU%). Here PVSCSI also wins on throughput, so it\u0026rsquo;s a clean win. 4K random write: 31% better efficiency on PVSCSI, and nearly equivalent throughput. This is the original promise of the paravirtual driver: eliminate the emulation overhead, get more I/O per CPU cycle. The CPU numbers confirm it works. Whether the throughput trade-off on large-block writes matters depends on your workload.\nThe main takeaways # SIOC matters more than controller type. Without SIOC, a single VM in a five-VM cluster pulled 5.6× more bandwidth than its neighbors. With SIOC, the variance compressed to roughly 1.4×. If you\u0026rsquo;re running multiple storage-intensive VMs on a shared datastore and SIOC is disabled, you have a fairness problem regardless of what controller you\u0026rsquo;re using.\nPVSCSI\u0026rsquo;s real advantage is CPU efficiency, not raw throughput. The throughput differences between controllers are workload-specific and not consistently in PVSCSI\u0026rsquo;s favor. What is consistent: PVSCSI burns 30–60% less CPU per IOPS across every test. On a host running many storage-heavy VMs, that overhead adds up. If you\u0026rsquo;re CPU-constrained, PVSCSI matters even when the IOPS numbers look similar.\nThe iodepth sweep is diagnostic, not just informational. The queue saturation behavior (IOPS peak then drop as depth increases) tells you where your storage ceiling is. gus_fio_2 peaked at depth 32 and degraded from there — that\u0026rsquo;s the point where adding more queue depth hurts rather than helps.\nThe practical recommendation: Enable SIOC on every shared datastore — the fairness argument is unambiguous regardless of controller. For controller choice, PVSCSI is the right default: the CPU efficiency win is consistent across all workloads, and for most mixed or read-heavy workloads the throughput numbers are equivalent or better. The one case worth benchmarking before committing is large-block bulk writes (backups, ETL, video ingest) — the regression there was significant enough that if that\u0026rsquo;s your primary workload, you should verify it on your specific storage before assuming PVSCSI is the right call.\nVM placement inside a datastore has real effects. gus_fio_3 and gus_fio_5 consistently underperformed gus_fio_1 and gus_fio_2 across every configuration, including with SIOC. If VMs on the same datastore are getting systematically different performance, suspect disk extent or spindle layout before blaming the controller or the guest.\nAll tests used fio_benchmark — the benchmark script covered in the previous post. 300-second runtime per test, libaio, direct I/O, no filesystem caching.\nRaw results from all five VMs across all four configurations are available for download if you want to verify the numbers or run your own analysis: results_fio.txt.\n","date":"2 April 2026","externalUrl":null,"permalink":"/posts/pvscsi-vs-lsi-sioc/","section":"Posts","summary":"","title":"PVSCSI vs LSI on vSphere: does the controller choice actually matter?","type":"posts"},{"content":"","date":"2 April 2026","externalUrl":null,"permalink":"/tags/sioc/","section":"Tags","summary":"","title":"Sioc","type":"tags"},{"content":"","date":"2 April 2026","externalUrl":null,"permalink":"/categories/storage/","section":"Categories","summary":"","title":"Storage","type":"categories"},{"content":"","date":"2 April 2026","externalUrl":null,"permalink":"/tags/storage/","section":"Tags","summary":"","title":"Storage","type":"tags"},{"content":"Every storage troubleshooting session starts the same way: you SSH into a box, pull up your notes (or browser history), and reconstruct the same handful of fio commands you\u0026rsquo;ve run a hundred times before. Sequential read, sequential write, 4K random read, mixed R/W — each with slightly different block sizes, queue depths, and job counts depending on what you\u0026rsquo;re hunting.\nThat repetitive scaffolding is exactly what fio_benchmark eliminates.\nWARNING — THIS SCRIPT WRITES DIRECTLY TO THE BLOCK DEVICE YOU SPECIFY.\nPointing it at any disk that contains data — your OS drive, a mounted volume, an in-use datastore — will corrupt or destroy that data, immediately and without any confirmation prompt. There is no undo.\nBefore running, confirm your target with lsblk or fdisk -l. If /dev/sda is your system disk, /dev/sdb is your VM datastore, or you are not 100% certain what is on the device, do not run this script against it. Use a blank, dedicated test disk or a test file on a disposable filesystem (-t /mnt/testdir/testfile).\nWhat it is (and what it isn\u0026rsquo;t) # fio_benchmark is a Bash wrapper around fio. It doesn\u0026rsquo;t replace fio — it doesn\u0026rsquo;t try to. fio is already the right tool. What it does is compile the test profiles most sysadmins and storage professionals run separately into a single repeatable execution, with consistent parameters and structured output you can actually compare across systems.\nThe target audience is straightforward: if you\u0026rsquo;re a sysadmin or storage engineer who needs to quickly characterize a device — whether it\u0026rsquo;s a new NVMe on a bare-metal server, a SAN LUN, or a VM disk you suspect is being throttled — this gets you there without building the test matrix from scratch every time.\nThe 9 profiles it runs # All tests use --direct=1, --ioengine=libaio, and --group_reporting. Random tests additionally disable repeat patterns (--randrepeat=0 --norandommap) to avoid cache-warming artifacts.\n# Test Pattern Block Size Jobs iodepth Size 1 seqread Sequential Read 8k 8 4 1G 2 seqwrite Sequential Write 32k 4 4 2G 3 seq128kread Sequential Read (max BW) 128k 8 4 2G 4 seq128kwrite Sequential Write (max BW) 128k 4 4 2G 5 randread Random Read 8k 16 8 1G 6 randwrite Random Write 64k 8 8 512m 7 rand4kread 4K Random Read 4k 16 8 1G 8 rand4kwrite 4K Random Write 4k 8 8 1G 9 randrw Random R/W (90% read) 16k 8 8 1G For each test, it collects IOPS, bandwidth (MiB/s), average latency, latency standard deviation, p50/p99/p99.9 percentiles, and CPU utilization. It also calculates an IOPS-per-CPU-percent efficiency ratio — useful when you\u0026rsquo;re comparing a bare-metal result against a VM running on the same hardware and want to quantify hypervisor overhead.\nInstallation # Requirements: Bash 4.4+, fio 3.0+, POSIX awk. No jq.\ngit clone https://github.com/9llo/fio_benchmark.git chmod +x fio_benchmark/fio_benchmark.sh Running it # The simplest invocation against a block device (requires root):\nsudo ./fio_benchmark.sh -t /dev/sdb If you want a dry-run first to verify the setup without touching the disk:\n./fio_benchmark.sh -t /dev/sdb -n No arguments drops you into interactive mode, which prompts for the target and runtime.\nOutput format is plain text by default. For scripting or long-term tracking, use JSON or CSV — progress and status messages always go to stderr, so the structured output on stdout stays clean for redirection:\nsudo ./fio_benchmark.sh -t /dev/sdb -o json \u0026gt; results.json sudo ./fio_benchmark.sh -t /dev/sdb -o csv \u0026gt; results.csv Runtime per test defaults to 600 seconds. For a faster pass during initial triage you can shorten it:\nsudo ./fio_benchmark.sh -t /dev/sdb -r 60 The two sweeps that actually find bottlenecks # The standard profiles give you a baseline. The sweeps are where you diagnose why a device behaves the way it does.\niodepth sweep (-s) # Runs randread with bs=4k, numjobs=1, and steps through iodepth values of 1, 2, 4, 8, 16, 32, 64, 128.\nsudo ./fio_benchmark.sh -t /dev/sdb -s The IOPS curve tells you where the device\u0026rsquo;s internal parallelism saturates. A consumer NVMe typically plateaus around iodepth 8–16. An enterprise SSD or a SAN LUN with a deep command queue can keep climbing well past 32. If you see IOPS plateau early and then degrade, that\u0026rsquo;s usually a firmware or queue management issue worth investigating.\nRW-mix sweep (-m) # Runs randrw with bs=4k, numjobs=8, iodepth=8, stepping write percentage from 0% to 100% in 10% increments.\nsudo ./fio_benchmark.sh -t /dev/sdb -m This maps the write-penalty curve. On a healthy NVMe, IOPS should degrade fairly gradually as write % increases. On a heavily overprovisioned SSD near capacity, or a thin-provisioned datastore being over-committed, the curve drops sharply as writes start triggering garbage collection and write amplification. If you\u0026rsquo;re running a database and wondering why performance cliffs under write-heavy workloads, this sweep will show it.\nOutput example (text) # Device: /dev/sdb | Samsung SSD 870 QVO | SSD | 1000GB | 512B blocks | mq-deadline ┌──────────────────┬──────────┬──────────┬────────────┬────────────┬────────────┬────────────┬────────────┬───────┬───────┬──────────────┐ │ Test │ IOPS │ BW MB/s │ Lat avg │ Lat std │ p50 │ p99 │ p99.9 │ CPU u │ CPU s │ IOPS/CPU% │ ├──────────────────┼──────────┼──────────┼────────────┼────────────┼────────────┼────────────┼────────────┼───────┼───────┼──────────────┤ │ seqread │ 52341 │ 409.7 │ 0.61 ms │ 0.12 ms │ 0.58 ms │ 0.94 ms │ 1.24 ms │ 3.1% │ 2.1% │ 10018.7 │ │ seqwrite │ 14832 │ 462.9 │ 1.08 ms │ 0.31 ms │ 0.99 ms │ 1.97 ms │ 3.51 ms │ 2.2% │ 1.8% │ 3707.8 │ ... When to use it # Qualifying new storage hardware before it enters production Comparing VM disk performance against bare-metal to measure virtualization overhead Checking whether a \u0026ldquo;slow storage\u0026rdquo; complaint from an application team is real or a red herring Building a baseline before and after a firmware update or storage configuration change Investigating an SSD that feels slower than expected after months of use (wear and write amplification) It won\u0026rsquo;t tell you why something is slow on its own — that still takes interpretation. But having the full test matrix, with consistent parameters, in under an hour, is a better starting point than assembling it by hand every time.\nA word on how this was built # Full disclosure: a good chunk of the iteration on this script was done with Claude Code as a pair programmer.\nThe domain knowledge — which tests matter, what block sizes to use for which workload profile, why the iodepth sweep is more useful than just running a single queue depth, how to parse fio\u0026rsquo;s JSON output without pulling in jq as a dependency — that all came from years of actual storage troubleshooting. Claude didn\u0026rsquo;t know that randrepeat=0 matters for avoiding cache-warming artifacts, or that an IOPS-per-CPU efficiency ratio is useful specifically in VM environments. That context had to be fed in.\nWhat it did accelerate: the Bash boilerplate nobody enjoys writing. The argument parser, the cleanup traps, the timeout wrapper around each fio invocation, the awk pass that extracts percentile fields from JSON without a dependency. Things that are tedious and error-prone to get right, but not intellectually interesting once you know what they need to do.\nThe honest summary: it took a task that would\u0026rsquo;ve been a full weekend of intermittent hacking and compressed it into a focused evening. The thinking didn\u0026rsquo;t get outsourced — it got unblocked.\nAI isn\u0026rsquo;t going to replace sysadmins or storage engineers. It has no idea what your SAN topology looks like, why your latency spikes at 3am, or what \u0026ldquo;it was fine before the firmware update\u0026rdquo; actually means in context. But if you can describe what you want precisely, it\u0026rsquo;s a remarkably good collaborator for the parts of the job that are just execution. The people who figure out how to use it well aren\u0026rsquo;t going to be replaced — they\u0026rsquo;re going to get more done.\nThe repository is at github.com/9llo/fio_benchmark. Issues and PRs are open.\n","date":"20 March 2026","externalUrl":null,"permalink":"/posts/fio_benchmark/","section":"Posts","summary":"","title":"fio_benchmark: A wrapper to stop writing the same fio commands over and over","type":"posts"},{"content":"","date":"20 March 2026","externalUrl":null,"permalink":"/tags/linux/","section":"Tags","summary":"","title":"Linux","type":"tags"},{"content":"","date":"20 March 2026","externalUrl":null,"permalink":"/tags/sysadmin/","section":"Tags","summary":"","title":"Sysadmin","type":"tags"},{"content":"","date":"6 March 2026","externalUrl":null,"permalink":"/categories/ai/","section":"Categories","summary":"","title":"AI","type":"categories"},{"content":"","date":"6 March 2026","externalUrl":null,"permalink":"/tags/amd/","section":"Tags","summary":"","title":"Amd","type":"tags"},{"content":"Running local Large Language Models (LLMs) on a CPU is slow. If your homelab server has an AMD Ryzen APU, you can use that integrated GPU to speed up inference.\nRunning everything inside a lightweight Proxmox LXC container instead of a full VM saves resources. However, unprivileged container permissions and AMD driver quirks often break the setup.\nMy hardware:\nCPU: AMD Ryzen 5 4600G with Radeon Graphics (1 Socket) Memory: 128GB OS: Proxmox Virtual Environment 9.1.6 This is for testing, learning, and self-hosting lightweight models on homelab hardware, not building a massive AI workstation.\nHere is how to pass your AMD APU through to an LXC container, install Docker, and deploy a hardware-accelerated AI stack with Ollama and Open WebUI.\nPhase 1: Prepare the Proxmox Host # All commands in this phase must be run on the Proxmox Host shell (root@pve1).\n1. Install AMD Firmware (The Non-Free Repository) # Proxmox (based on Debian) doesn\u0026rsquo;t include the required AMD graphics firmware by default. We need to enable the non-free firmware repositories to install the package.\nnano /etc/apt/sources.list Ensure your primary Debian repository lines include non-free and non-free-firmware. It should look something like this:\ndeb http://deb.debian.org/debian trixie main contrib non-free non-free-firmware deb http://security.debian.org/debian-security trixie-security main contrib non-free non-free-firmware deb http://deb.debian.org/debian trixie-updates main contrib non-free non-free-firmware (Note: If you are on an older version of Proxmox, replace trixie with bookworm).\nUpdate your packages and install the firmware:\napt update apt install -y firmware-amd-graphics 2. Configure GRUB for IOMMU # First, enable IOMMU so the host can access the GPU hardware directly.\nnano /etc/default/grub Modify the GRUB_CMDLINE_LINUX_DEFAULT line to look exactly like this (remove any video=efifb:off or nomodeset flags if they exist):\nGRUB_CMDLINE_LINUX_DEFAULT=\u0026#34;quiet amd_iommu=on iommu=pt\u0026#34; Update GRUB to apply:\nupdate-grub 3. Disable VFIO \u0026ldquo;Hijacking\u0026rdquo; # If you\u0026rsquo;ve previously set up VM passthrough, you might have blacklisted the GPU drivers. We need the host to use them for LXC.\nnano /etc/modprobe.d/vfio.conf Comment out (add # to the start) every line so it looks like this:\n# options vfio-pci ids=1002:1636... # softdep amdgpu pre: vfio-pci # softdep snd_hda_intel pre: vfio-pci 4. Force the amdgpu Module to Load # Ensure the host loads the AMD drivers at boot.\nnano /etc/modules Add these lines to the bottom:\nvfio vfio_iommu_type1 vfio_pci vfio_virqfd amdgpu 5. Check for Hidden AMD Blacklists # Before rebooting, ensure Proxmox isn\u0026rsquo;t actively blocking the driver. Run the following check:\ngrep -ir \u0026#34;amdgpu\u0026#34; /etc/modprobe.d/ If you see a file like blacklist-amdgpu.conf, you will need to rename it, or the driver will never load regardless of your configuration in /etc/modules:\nmv /etc/modprobe.d/blacklist-amdgpu.conf /etc/modprobe.d/blacklist-amdgpu.conf.bak 6. Apply Changes and Reboot # Regenerate the boot images and restart your Proxmox server.\nupdate-initramfs -u -k all reboot 7. Verify the Drivers Loaded # Once the host is back up, check for the GPU device files:\nls -l /dev/dri You should see card0 and renderD128.\nPhase 2: Fix Container Permissions (The Udev Fix) # By default, unprivileged LXC containers map the root user to a high-numbered \u0026ldquo;nobody\u0026rdquo; user on the host. If you pass the GPU through now, the container will be denied access. We fix this with a Udev rule on the host.\nRun this on the Proxmox Host:\n# Create the rule to make the GPU globally readable/writable echo \u0026#39;KERNEL==\u0026#34;renderD128\u0026#34;, MODE=\u0026#34;0666\u0026#34;\u0026#39; \u0026gt; /etc/udev/rules.d/99-render.rules echo \u0026#39;KERNEL==\u0026#34;card0\u0026#34;, MODE=\u0026#34;0666\u0026#34;\u0026#39; \u0026gt;\u0026gt; /etc/udev/rules.d/99-render.rules # Apply the rule immediately udevadm control --reload-rules \u0026amp;\u0026amp; udevadm trigger Verify the permissions changed (look for crw-rw-rw-):\nls -l /dev/dri/renderD128 Phase 3: Create and Configure the LXC Container # 1. Provision the Container # Create a new unprivileged LXC container using the Proxmox CLI. For the AI stack, Ubuntu 24.04 is the recommended template. First, update your template cache and download the image:\npveam update pveam download local ubuntu-24.04-standard_24.04-2_amd64.tar.zst Create the container (we use ID 100 as an example), assigning 4 CPU cores, 4GB of RAM, and 30GB of storage:\npct create 100 local:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst \\ --unprivileged 1 \\ --hostname local-ai \\ --cores 6 \\ --memory 49152 \\ --swap 0 \\ --rootfs local-lvm:30 \\ --net0 name=eth0,bridge=vmbr0,ip=10.1.99.16/16,gw=10.1.0.1,tag=99 Do not start the container yet.\n2. Get Host Group IDs (Optional) # Since the Udev rule from Phase 2 makes the GPU world-writable (0666), matching group IDs isn\u0026rsquo;t strictly necessary for the container to boot. However, keeping them aligned remains a solid structural best practice. Find the IDs for the render and video groups on your host:\ngetent group render | cut -d: -f3 # Example: 104 getent group video | cut -d: -f3 # Example: 44 3. Enable LXC Nesting # In the Proxmox Web UI, select your container -\u0026gt; Options -\u0026gt; Features. Check the box for Nesting. This is mandatory for running Docker inside LXC.\n4. Edit the LXC Config # Find your container ID (e.g., 100). Edit its configuration file on the Host:\nnano /etc/pve/lxc/100.conf Append the following lines to pass the hardware through (update the GIDs if yours differed from 104 and 44):\n# GPU Passthrough lxc.cgroup2.devices.allow: c 226:0 rwm lxc.cgroup2.devices.allow: c 226:128 rwm lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file Start (or restart) your LXC container.\nPhase 4: Deploying Ollama and Open WebUI with Docker Compose # Running discrete containers requires multiple commands and makes networking fragile. We define the AI stack using Docker Compose instead.\nLog into your LXC container console (pct enter 100 or SSH).\n1. Install Docker # First, grab the official script to install Docker (which includes the Compose plugin) on your fresh Ubuntu container:\napt update \u0026amp;\u0026amp; apt upgrade -y curl -fsSL https://get.docker.com | sh 2. Define the Stack Configuration # Create a new directory for the AI services and write the docker-compose.yml file:\nmkdir -p ~/ai-stack \u0026amp;\u0026amp; cd ~/ai-stack nano docker-compose.yml Paste the following YAML into the file:\nservices: # ------------------------------------- # 1. Reverse Proxy (Traefik) # ------------------------------------- traefik: image: traefik:v3.0 container_name: traefik restart: unless-stopped command: - \u0026#34;--api.insecure=true\u0026#34; # Enables the Traefik dashboard on port 8080 (optional) - \u0026#34;--providers.docker=true\u0026#34; - \u0026#34;--providers.docker.exposedbydefault=false\u0026#34; # Security: deny mapping unless explicitly labeled # Define both HTTP (80) and HTTPS (443) entrypoints - \u0026#34;--entrypoints.web.address=:80\u0026#34; - \u0026#34;--entrypoints.websecure.address=:443\u0026#34; # Automatically redirect all HTTP traffic to HTTPS (websecure) - \u0026#34;--entrypoints.web.http.redirections.entrypoint.to=websecure\u0026#34; - \u0026#34;--entrypoints.web.http.redirections.entrypoint.scheme=https\u0026#34; ports: - \u0026#34;80:80\u0026#34; # Standard HTTP entrypoint - \u0026#34;443:443\u0026#34; # Secure HTTPS entrypoint - \u0026#34;8080:8080\u0026#34; # Traefik Dashboard volumes: - \u0026#34;/var/run/docker.sock:/var/run/docker.sock:ro\u0026#34; # Secure read-only access to Docker events networks: - proxy-net logging: driver: \u0026#34;json-file\u0026#34; options: max-size: \u0026#34;10m\u0026#34; max-file: \u0026#34;3\u0026#34; # ------------------------------------- # 2. LLM Backend (Ollama) # ------------------------------------- ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped user: \u0026#34;root\u0026#34; # Force hardware permissions privileged: true # Bypass Docker AppArmor blocking IO controls to /dev/dri networks: - ai-net devices: - /dev/dri:/dev/dri volumes: - ollama_data:/root/.ollama environment: - OLLAMA_HOST=0.0.0.0 - OLLAMA_VULKAN=1 healthcheck: test: [\u0026#34;CMD\u0026#34;, \u0026#34;ollama\u0026#34;, \u0026#34;list\u0026#34;] interval: 10s timeout: 5s retries: 5 start_period: 15s deploy: resources: limits: memory: 48G logging: driver: \u0026#34;json-file\u0026#34; options: max-size: \u0026#34;20m\u0026#34; max-file: \u0026#34;3\u0026#34; # ------------------------------------- # 3. Frontend UI (Open-WebUI) # ------------------------------------- open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped networks: - ai-net # Internal network to reach Ollama - proxy-net # Edge network facing Traefik volumes: - open_webui_data:/app/backend/data environment: - OLLAMA_BASE_URL=http://ollama:11434 depends_on: ollama: condition: service_healthy # Will not try to start before Ollama\u0026#39;s API responds # Principle of Least Privilege cap_drop: - ALL deploy: resources: limits: memory: 2G logging: driver: \u0026#34;json-file\u0026#34; options: max-size: \u0026#34;10m\u0026#34; max-file: \u0026#34;3\u0026#34; labels: - \u0026#34;traefik.enable=true\u0026#34; # Tell Traefik exactly which network to use to reach this container! - \u0026#34;traefik.docker.network=ai-stack_proxy-net\u0026#34; # Catch-all rule: Allows ANY IP or domain to reach Open-WebUI - \u0026#34;traefik.http.routers.openwebui.rule=PathPrefix(`/`)\u0026#34; # Route traffic through HTTPS and generate a self-signed certificate - \u0026#34;traefik.http.routers.openwebui.entrypoints=websecure\u0026#34; - \u0026#34;traefik.http.routers.openwebui.tls=true\u0026#34; - \u0026#34;traefik.http.services.openwebui.loadbalancer.server.port=8080\u0026#34; # Traefik Security Middleware - \u0026#34;traefik.http.middlewares.secure-headers.headers.browserXssFilter=true\u0026#34; - \u0026#34;traefik.http.middlewares.secure-headers.headers.contentTypeNosniff=true\u0026#34; - \u0026#34;traefik.http.middlewares.secure-headers.headers.forceSTSHeader=true\u0026#34; - \u0026#34;traefik.http.routers.openwebui.middlewares=secure-headers\u0026#34; # ------------------------------------- # Networks \u0026amp; Volumes # ------------------------------------- networks: ai-net: driver: bridge # Secure internal layer proxy-net: driver: bridge # External ingress layer volumes: ollama_data: open_webui_data: Why this architecture matters # Running raw docker run commands works for a quick demo, but breaks down on a permanent homelab server. Here is why we structured the docker-compose.yml this way:\nAir-gapped LLM API: Ollama is bound exclusively to the internal ai-net bridge. There are zero exposed host ports. The only way to talk to the model is through the Open WebUI container over the isolated network. Dynamic Ingress with Traefik: Instead of statically binding host ports, Traefik acts as a smart reverse proxy. It routes inbound traffic on ports 80/443 directly to Open WebUI using Docker labels, forces automatic HTTPS redirection, and injects strict security headers (browserXssFilter, contentTypeNosniff). The \u0026ldquo;Memory Backoff\u0026rdquo; Trap: Because the APU shares system RAM, the Vulkan driver imposes a strict limit on single memory allocations. If you try to load a model that is too large, the GPU fails the initial allocation and \u0026ldquo;backs off,\u0026rdquo; dumping those layers onto the CPU. If you see 0% GPU usage during inference, this is why. By bumping the LXC and Ollama memory limit up (in our case, to 48GB), we provide the necessary headroom for the Vulkan buffer. The latest vs rocm Tag: Many AMD users defensively pull the rocm image tag. Do not do this for APUs. On a Ryzen APU, the ROCm abstraction often fails to initialize the integrated graphics and silently falls back to CPU execution. For APU Vulkan acceleration, the latest image simply works. Principle of Least Privilege: We stripped every unnecessary Linux root capability from the Open WebUI container (cap_drop: ALL). If an exploit surfaces in the frontend, the blast radius is confined. Solving the 500 Error Crash: Normally, Open WebUI boots faster than Ollama can initialize the AMD GPU drivers. When the UI tries to connect and fails, it throws a permanent 500: Internal Error crash. We fixed this by building a native CMD ollama list health check that probes the API. Traefik and WebUI structurally wait for Ollama to become healthy before attempting connection. 3. Deploy the Stack # Spin up the entire environment in the background:\ndocker compose up -d 4. Verify Hardware Acceleration # Verify Ollama detects the AMD graphics before downloading any multi-gigabyte models.\ndocker logs ollama | grep \u0026#34;AMD\u0026#34; Expected output: ggml_vulkan: Found AMD Radeon Graphics\nBonus: How to Monitor GPU Usage (radeontop) # If you run inferences via CLI, you can monitor APU metrics in real-time using radeontop, an AMD alternative to nvidia-smi.\nSince you configured the container to access /dev/dri, you can run this directly from inside the LXC environment.\nFirst, install the utility:\napt install -y radeontop Next, simply launch it to view rendering activity, VRAM allocations, and shader clock speeds:\nradeontop When you generate a response in Open WebUI, the Graphics pipe utilization briefly spikes to 100% as the GPU processes tokens.\nThe Final Result # Once it\u0026rsquo;s running, you have a local, private AI environment. Hardware acceleration makes tasks like summarizing full articles fast.\nAccessing Open WebUI # You can access the web interface by navigating to https://10.1.99.16 (or whatever IP you assigned to your LXC container) in your browser. Since Traefik handles the routing, it automatically answers on port 443 with a self-signed certificate.\nHere is Open WebUI summarizing a link using the APU:\n","date":"6 March 2026","externalUrl":null,"permalink":"/posts/AMD_APU_to_LXC/","section":"Posts","summary":"","title":"AMD APU Passthrough to a Proxmox LXC for Local AI","type":"posts"},{"content":"","date":"6 March 2026","externalUrl":null,"permalink":"/tags/docker/","section":"Tags","summary":"","title":"Docker","type":"tags"},{"content":"","date":"6 March 2026","externalUrl":null,"permalink":"/tags/gpu-passthrough/","section":"Tags","summary":"","title":"Gpu-Passthrough","type":"tags"},{"content":"","date":"6 March 2026","externalUrl":null,"permalink":"/categories/homelab/","section":"Categories","summary":"","title":"Homelab","type":"categories"},{"content":"","date":"6 March 2026","externalUrl":null,"permalink":"/tags/llm/","section":"Tags","summary":"","title":"Llm","type":"tags"},{"content":"","date":"6 March 2026","externalUrl":null,"permalink":"/tags/lxc/","section":"Tags","summary":"","title":"Lxc","type":"tags"},{"content":"","date":"6 March 2026","externalUrl":null,"permalink":"/tags/ollama/","section":"Tags","summary":"","title":"Ollama","type":"tags"},{"content":"","date":"6 March 2026","externalUrl":null,"permalink":"/tags/open-webui/","section":"Tags","summary":"","title":"Open-Webui","type":"tags"},{"content":"","date":"6 March 2026","externalUrl":null,"permalink":"/categories/proxmox/","section":"Categories","summary":"","title":"Proxmox","type":"categories"},{"content":"","date":"6 March 2026","externalUrl":null,"permalink":"/tags/proxmox/","section":"Tags","summary":"","title":"Proxmox","type":"tags"},{"content":"","date":"6 March 2026","externalUrl":null,"permalink":"/tags/ryzen/","section":"Tags","summary":"","title":"Ryzen","type":"tags"},{"content":" Failed tasks in SDDC # Recenlty I faced a weird case with VMware, out of nowhere failed tasks started to show on my SDDC, intially I thought it was something related to password expired, but no, not this time.\nThe logs don\u0026rsquo;t lie # People lie, but the logs don\u0026rsquo;t. After a couple days working with VMware support and we didn\u0026rsquo;t seem to find any clue of what was happening, by accident I saw something on the SOS healthcheck that caught my attention, it was complaining that the SDDC had problems to connect to my host through API (port 443).\nHow to run a health check on SDDC\n/opt/vmware/sddc-support/sos --health-check --domain-name ALL --skip-cert-check | 21 | ESXi : testhost1.9tec.local | Ping status | GREEN | | | | API Connectivity status | RED | | 21 | ESXI : testhost1.9tec.local | svc-vcf-testhost1 | Jul 20, 2025 | Never | Never | GREEN | | | | root | - | - | - | Failed to get details | | 21 | ESXi : testhost1.9tec.local | NTP Status | YELLOW | | | | ESX Time | YELLOW | Dang, it reminded me of a situation I found not long ago, where my vSphere replication was causing the reverse proxy on ESXi to exhaust the sessions (the limit is 128). Reference: ESXi hosts go unresponsive on the vCenter Server with maximum connection exceeded for the envoy proxy.\nWell well, let\u0026rsquo;s take a look on the envoy.log on this particular host and see what\u0026rsquo;s going on\nless /var/log/envoy.log ## or less /var/log/envoy.log Search by the word \u0026ldquo;exceed\u0026rdquo; and you\u0026rsquo;ll see if there are any evidences of exceeding max connections\n2025-08-13T22:14:06.853Z In(166) envoy[2100238]: \u0026#34;2025-08-13T22:13:57.261Z warning envoy[2100869] [Originator@6876 sub=filter] [Tags: \u0026#34;ConnectionId\u0026#34;:\u0026#34;696461\u0026#34;] remote https connections exceed max allowed: 128\u0026#34; 2025-08-13T22:14:06.853Z In(166) envoy[2100238]: \u0026#34;2025-08-13T22:13:57.261Z warning envoy[2100869] [Originator@6876 sub=filter] [Tags: \u0026#34;ConnectionId\u0026#34;:\u0026#34;696461\u0026#34;] closing connection TCP\u0026lt;my_vcenter_IP:59956, my_esxi_IP:443\u0026gt;\u0026#34; 2025-08-13T22:14:06.853Z In(166) envoy[2100238]: \u0026#34;2025-08-13T22:13:57.261Z info envoy[2100869] [Originator@6876 sub=connection] [Tags: \u0026#34;ConnectionId\u0026#34;:\u0026#34;696461\u0026#34;] remote address:my_vcenter_IP:59956,TLS_error:|268435588:SSL routines:OPENSSL_internal:CLIENTHELLO_TLSEXT|268435646:SSL routines:OPENSSL_internal:PARSE_TLSEXT\u0026#34; The workaround # As the KB mentioned above says, there\u0026rsquo;s no fix for that, but a workaround that consists in disabling the healthcheck on the vSphere replication (that\u0026rsquo;s what is using the sessions on the ESXi hosts due to enhanced replication fancy checks)\n1 - Open a SSH session to the VRMS server on both the sites.\n2 - Open file /opt/vmware/hms/conf/hms-configuration.xml with a text editor`\nvi /opt/vmware/hms/conf/hms-configuration.xml 3 - Set schedule-health-checks to false\n4 - Restart HMS service on both sites\nsystemctl restart hms 5 - While configuring enhanced replication, skip the health check. Clicking the \u0026ldquo;Next\u0026rdquo; button will allow you to proceed with the replication configuration without performing the health check.\n","date":"18 August 2025","externalUrl":null,"permalink":"/posts/failed-tasks-in-sddc-5-2/","section":"Posts","summary":"","title":"Failed tasks in SDDC 5.2","type":"posts"},{"content":"","date":"18 August 2025","externalUrl":null,"permalink":"/tags/sddc/","section":"Tags","summary":"","title":"Sddc","type":"tags"},{"content":"","date":"18 August 2025","externalUrl":null,"permalink":"/categories/sddc/","section":"Categories","summary":"","title":"SDDC","type":"categories"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/categories/automation/","section":"Categories","summary":"","title":"Automation","type":"categories"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/categories/esxi/","section":"Categories","summary":"","title":"ESXi","type":"categories"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/tags/ks.cfg/","section":"Tags","summary":"","title":"Ks.cfg","type":"tags"},{"content":" Requirements # Linux machine (I\u0026rsquo;m using WSL running ubuntu 24.04 in this example) ESXi 8.0U3 ISO (it should work in older versions, but some parameters may be different) Key objectives # Spin up a temporary HTTP server that you can use in field for mass ESXi deployment Create a custom ESXi ISO file with boot.cfg edited to read ks.cfg from a HTTP server Explain the logic behin my scripts The custom ISO # Edit the BOOT.CFG # Download the latest ESXi ISO from Broadcom portal here\nMount that ISO file into your system and copy the content of the ISO to a directory inside of your linux machine (in my case is /home/novello/ESXi8/)\nOnce you have those files on a directory, is time to edit the file BOOT.CFG, responsible for specifying where is the kickstart file\nvi /home/novello/ESXi8/EFI/BOOT/BOOT.CFG and\nvi /home/novello/ESXi8/BOOT.CFG Edited lines: timeout: Removed the splash screen to skip the \u0026ldquo;loading\u0026rdquo; screen. kernelopt: Added the path to the ks.cfg file hosted on the web server. bootstate=0 title=Loading ESXi installer timeout=0 # Here I removed the timeout to not show the \u0026#34;loading\u0026#34; screen prefix= kernel=/b.b00 kernelopt=runweasel ks=http://192.168.100.32:8000/ks.cfg # Here I added the ks.cfg file to the boot options, loaded from a web server modules=/jumpstrt.gz --- /useropts.gz --- /features.gz --- /k.b00 --- /uc_intel.b00 --- /uc_amd.b00 --- /uc_hygon.b00 --- /procfs.b00 --- /vmx.v00 --- /vim.v00 --- /tpm.v00 --- /sb.v00 --- /s.v00 --- /atlantic.v00 --- /bcm_mpi3.v00 --- /bnxtnet.v00 --- /bnxtroce.v00 --- /brcmfcoe.v00 --- /cndi_igc.v00 --- /dwi2c.v00 --- /elxiscsi.v00 --- /elxnet.v00 --- /i40en.v00 --- /iavmd.v00 --- /icen.v00 --- /igbn.v00 --- /intelgpi.v00 --- /ionic_cl.v00 --- /ionic_en.v00 --- /irdman.v00 --- /iser.v00 --- /ixgben.v00 --- /lpfc.v00 --- /lpnic.v00 --- /lsi_mr3.v00 --- /lsi_msgp.v00 --- /lsi_msgp.v01 --- /lsi_msgp.v02 --- /mtip32xx.v00 --- /ne1000.v00 --- /nenic.v00 --- /nfnic.v00 --- /nhpsa.v00 --- /nipmi.v00 --- /nmlx5_cc.v00 --- /nmlx5_co.v00 --- /nmlx5_rd.v00 --- /ntg3.v00 --- /nvme_pci.v00 --- /nvmerdma.v00 --- /nvmetcp.v00 --- /nvmxnet3.v00 --- /nvmxnet3.v01 --- /pvscsi.v00 --- /qcnic.v00 --- /qedentv.v00 --- /qedrntv.v00 --- /qfle3.v00 --- /qfle3f.v00 --- /qfle3i.v00 --- /qflge.v00 --- /rdmahl.v00 --- /rste.v00 --- /sfvmk.v00 --- /smartpqi.v00 --- /vmkata.v00 --- /vmksdhci.v00 --- /vmkusb.v00 --- /vmw_ahci.v00 --- /bmcal.v00 --- /clusters.v00 --- /crx.v00 --- /drivervm.v00 --- /elx_esx_.v00 --- /btldr.v00 --- /dvfilter.v00 --- /esx_ui.v00 --- /esxupdt.v00 --- /tpmesxup.v00 --- /weaselin.v00 --- /esxio_co.v00 --- /infravis.v00 --- /loadesx.v00 --- /lsuv2_hp.v00 --- /lsuv2_in.v00 --- /lsuv2_ls.v00 --- /lsuv2_nv.v00 --- /lsuv2_oe.v00 --- /lsuv2_oe.v01 --- /lsuv2_sm.v00 --- /native_m.v00 --- /qlnative.v00 --- /trx.v00 --- /vcls_pod.v00 --- /vdfs.v00 --- /vds_vsip.v00 --- /vmware_e.v00 --- /hbrsrv.v00 --- /vsan.v00 --- /vsanheal.v00 --- /vsanmgmt.v00 --- /tools.t00 --- /xorg.v00 --- /gc.v00 --- /imgdb.tgz --- /basemisc.tgz --- /resvibs.tgz --- /esxiodpt.tgz --- /imgpayld.tgz build=8.0.3-0.0.24022510 updated=0 Compile the new files into a new custom ISO # Here\u0026rsquo;s where the magic happens, how do we compile all those previous extracted files into a new custom .ISO? Simple!\nParameters you need to change:\n-o: Path/name of the file you want to save. In this example /home/novello/ESXi8U3_custom.iso Path where the ISO was extracted, in my case /home/novello/ESXi8 genisoimage -relaxed-filenames -J -R -o /home/novello/ESXi8U3_custom.iso -b ISOLINUX.BIN -c BOOT.CAT -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e EFIBOOT.IMG -no-emul-boot /home/novello/ESXi8 ks.cfg (where the magic really happens) # Okay, so in the BOOT.CFG I informed that when the host boots up, it should read a kickstart file and use the logic in there to start configuring the ESXi. Key objectives ks.cfg # Load ESXi installation and accept the EULA Define root password Grab initial IP using vmnic0 and DHCP Enable SSH/Bash shell Executes python script that will configure definitive IP / Hostname on the host vmaccepteula rootpw VMware123! install --firstdisk --overwritevmfs network --bootproto=dhcp --device=vmnic0 reboot %firstboot --interpreter=busybox # Wait for the network to start sleep 5 # Enable SSH and ESXi Shell vim-cmd hostsvc/enable_ssh vim-cmd hostsvc/start_ssh vim-cmd hostsvc/enable_esx_shell vim-cmd hostsvc/start_esx_shell # Suppress ESXi Shell warning esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1 # Adjust the time zone cp /usr/share/zoneinfo/America/Sao_Paulo /etc/localtime # Download the host_config.py script and macs.csv file wget http://192.168.100.32:8000/host_config.py -O /tmp/host_config.py wget http://192.168.100.32:8000/macs.csv -O /tmp/macs.csv # Make the script executable # (Note: The script is already executable in the original code, but this is a good practice) chmod u+x /tmp/host_config.py # Run the host_config.py script /bin/python3 /tmp/host_config.py Key objectives of host_config.py # Python reads the macs.csv Based on the MAC address of vmnic0 it determines which IP/mask/gw/VLAN/hostname vmk0(management) will be configured import os import csv import subprocess CSVPATH = \u0026#34;/tmp/macs.csv\u0026#34; # Path to the CSV file # Get the MAC address of the host interfaces MAC = subprocess.check_output( \u0026#34;esxcli network ip interface list | grep \u0026#39;MAC Address\u0026#39; | awk \u0026#39;{print $3}\u0026#39;\u0026#34;, shell=True ) MACADDR = MAC.split() print(\u0026#34;MACs found:\u0026#34;) print(MACADDR) # Read the CSV file and find the matching MAC address with open(CSVPATH, \u0026#39;r\u0026#39;) as csvFile: csvReader = csv.reader(csvFile) next(csvReader) # Skip the header row for csvRow in csvReader: # csvRow format: hostname,mac,ip,netmask,gateway,vlan if MACADDR[0].decode(\u0026#39;utf-8\u0026#39;).lower() == csvRow[1].lower(): hostname = csvRow[0] ip = csvRow[2] netmask = csvRow[3] gateway = csvRow[4] vlan = csvRow[5] print(f\u0026#34;Configuring host {hostname} with IP {ip}\u0026#34;) os.system(f\u0026#34;esxcli system hostname set --fqdn={hostname}\u0026#34;) os.system(f\u0026#34;esxcli network ip interface ipv4 set --interface-name=vmk0 --ipv4={ip} --netmask={netmask} --type=static\u0026#34;) os.system(f\u0026#34;esxcfg-route {gateway}\u0026#34;) os.system(f\u0026#34;esxcfg-vswitch -p \u0026#39;Management Network\u0026#39; -v {vlan} vSwitch0\u0026#34;) What macs.csv looks like\nhostname,mac,ip,netmask,gateway,vlan esxi01,00:50:56:11:22:33,10.1.99.101,255.255.255.0,10.1.99.1,99 esxi02,00:50:56:44:55:66,10.1.99.102,255.255.255.0,10.1.99.1,99 Spin up the HTTP server # I will use the same WSL instance I used to compile the ISO to serve as HTTP server I created a separate folder that will contain ks.cfg macs.csv and host_config.py python3 -m http.server -d /home/novello/http On your browser, navigate to http://\u0026lt;WSL_instance_IP\u0026gt;:8000, you should see something similar to And that\u0026rsquo;s all, we\u0026rsquo;re ready to start our ESXi installation and it should configure the basics for us automatically.\nInstalled # For testing purposes I created a VM into VMware workstation and customized the MAC address to match esxi02 of our macs.csv References # VMware Documentation on ks.cfg\nVMware Boot Options Guide\nWilliam Lam\n","date":"28 April 2025","externalUrl":null,"permalink":"/posts/ZTP_ESXi_installation/","section":"Posts","summary":"","title":"ZTP ESXi installation","type":"posts"},{"content":"","date":"23 April 2025","externalUrl":null,"permalink":"/categories/general/","section":"Categories","summary":"","title":"General","type":"categories"},{"content":"","date":"23 April 2025","externalUrl":null,"permalink":"/tags/general/","section":"Tags","summary":"","title":"General","type":"tags"},{"content":" Welcome 🤖🛸 # Hey there! This is my first post on the blog — a project that’s been sitting in my backlog for a while. I finally found the time to bring it to life 🚀\nIf you\u0026rsquo;re here, you can expect posts about VMware, Proxmox, Linux, automation, and plenty of homelab experiments 🛠️\nFeel free to leave a comment or suggestion — I’d love to hear what you’d like to see next!\n","date":"23 April 2025","externalUrl":null,"permalink":"/posts/Awakening/","section":"Posts","summary":"","title":"There has been an awakening!","type":"posts"},{"content":"Hello! I\u0026rsquo;m Gustavo Novello, a Cloud Infrastructure Specialist with a strong passion for virtualization, Linux, and open-source technologies. I bring extensive experience in managing large-scale VMware environments.\nMy professional journey is rooted in building resilient, scalable, and secure infrastructure solutions. I thrive on optimizing complex systems and enabling seamless cloud operations that support business agility and innovation.\nBeyond my technical expertise, I\u0026rsquo;m committed to continuous learning and sharing knowledge within the tech community. Whether it\u0026rsquo;s through collaboration, mentorship, or contributing to open-source projects, I believe in the power of collective growth.\nFeel free to connect with me on LinkedIn to explore potential collaborations or to discuss the latest in cloud infrastructure and virtualization.\n","externalUrl":null,"permalink":"/about/","section":"From homelab to hybrid cloud","summary":"","title":"About","type":"page"},{"content":"","externalUrl":null,"permalink":"/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"}]