Add guest debugging support #81

AlexAltea · 2018-07-22T14:17:05Z

This implements the debugging features proposed at #66. More information and questions can be found on that thread.

Motivation

Guest debugging is extremely relevant to debugging bootloaders/microkernels, or in my case, kernels that do not include debugging backends. Aside from the intrinsic value of this feature, it's also really beneficial to developers/researchers as neither WHPX (Windows) nor HVF (macOS) support guest debugging and while they remain closed-source this will be probably hard to change.

Design

Interface

Adding a HAX_VCPU_IOCTL_DEBUG ioctl (i.e. QEMU-to-HAXM) with a structure containing:
- Flags to enable/disable debugging and specific features:
  This is meant to manage the guest state accordingly: e.g. if guest debugging via hardware breakpoints is enabled we should protect the guest drN values in the exit_dr_access handler. Additionally, to know where to redirect #BP, e.g. if software breakpoints are enabled forward to QEMU, otherwise forward to VM.
- Values for debugging registers: dr0, dr1, dr2, dr3, dr6, dr7:
  They already are a self-contained way of describing hardware breakpoints, so no need to wrap that information in another layer of abstraction.
Adding a HAX_EXIT_DEBUG exit (i.e. HAXM-to-QEMU) with a structure containing:
- Value of: rip: For software breakpoint events.
- Value of: dr6, dr7: For hardware breakpoint events and single-stepping.

Features

Software breakpoints are managed by QEMU as an array of objects containing an address and the original byte at that instruction before patching int 3h (0xCC). Their lifecycle is:
1. GDB: Sends break * command.
2. QEMU: Creates/initializes hax_sw_brakpoint object and patches instruction with int 3h.
3. HAXM: Runs the virtual machine, and returns to QEMU after hitting #BP.
4. QEMU: Returns control to GDB.
5. GDB: Sends continue command.
6. QEMU: Reverts the instruction patch and single-steps.
7. HAXM: Executes one instruction and returns.
8. QEMU: Reapplies the instruction patch and resumes execution.
Hardware breakpoints are specified by directly writing into the guest drN registers through the previously described ioctl. Their lifecycle is:
1. GDB: Sends hbreak * command.
2. QEMU: Prepares the guest dr{0,1,2,3,7} values, and sends them via ioctl.
3. HAXM: Loads the drN into the guest. When asked to runs the virtual machine, dr{0,1,2,3,6} registers are saved/loaded manually, and dr7 automatically via VMCS, to avoid side-effects on the host. Returns to QEMU after hitting #DB, providing the value for dr6.
4. QEMU: With the information obtained from the dr6 it can identify the breakpoint corresponding to the hardware breakpoint. Then, returns control to GDB.
5. GDB: Sends continue command.
6. QEMU: Reverts the hardware breakpoint and single-steps.
7. HAXM: Executes one instruction and returns.
8. QEMU: Reapplies the hardware breakpoint and resumes execution.
Single-stepping is accomplished with the trap flag.
1. GDB: Sends si or ni command.
2. QEMU: Enables the HAX_DEBUG_STEP debug control flag with the previous ioctl.
3. HAXM: Enabled the Trap Flag (TF) in the FLAGS register. Runs the virtual machine, and returns to QEMU after hitting #DB, caused by the trap flag.
4. QEMU: Disables the HAX_DEBUG_STEP debug control flag with the previous ioctl.

Documentation

Although the description above might serve as a rough overview to review this patch, I haven't properly documented the interface changes on API.md. We can either leave that for a future PR, or if you want that already, let me know if you agree with the current proposal and I'll start documenting everything right away.

Samples

Since this patch adds a new feature for userland applications, we probably should provide some example. QEMU's HAXM accelerator seems to be the reference implementation, so I've added the debugging features to my QEMU fork at AlexAltea/orbital-qemu@77d61c7. Please feel free to upstream these changes to QEMU if you consider it appropriate.

raphaelning · 2018-07-23T10:54:14Z

Thanks a lot for implementing this massive feature and writing the nice cover page! It seems you've got single stepping working, which is great.

Sorry I didn't have time to try or review this PR today due to other tasks, but I'll try to give you some feedback tomorrow. Just two quick questions:

I mentioned in Guest debugging support #66 a use case where the user wants to debug guest user-mode code by running the debugger (e.g. gdb) from within the guest (e.g. Ubuntu). Is that supported now? (It's okay if it isn't--I'm just curious.)
Are (hardware) watchpoints supported? (Again, just curious.)

AlexAltea · 2018-07-23T11:22:05Z

I mentioned in #66 a use case where the user wants to debug guest user-mode code by running the debugger (e.g. gdb) from within the guest (e.g. Ubuntu). Is that supported now?

It works, as long as you don't enable the HAX_DEBUG_ENABLE-flag via the HAX_VCPU_IOCTL_DEBUG ioctl. This happens because HAXM-powered debugging intercepts #DB and #BP exceptions and modifies the guest drN registers, which will cause problems with guest debuggers.

By disabling the HAX_DEBUG_ENABLE-flag, this feature won't produce any side effects on guests.
In summary: guests can only be debugged from either host or guest, but not both simultaneously. I suppose that's not a big limitation (unless you want to debug kernel debuggers 😄).

Are (hardware) watchpoints supported?

On the HAXM-side they are supported, since this feature just overrides guest drN registers.
On the QEMU-side, they are implemented but I haven't tested them yet. I'll report back ~~tomorrow~~ this weekend.

raphaelning · 2018-07-24T10:21:24Z

core/vcpu.c

@@ -1132,17 +1132,35 @@ void vcpu_save_host_state(struct vcpu_t *vcpu)
    save_host_msr(vcpu);

    hstate->hcr2 = get_cr2();
+    hstate->dr0 = get_dr0();


Most of the time, the vCPU is not in debugging mode (!(vcpu->debug_control & HAX_DEBUG_ENABLE)), so we don't need to save/load debug registers. At least, we can ignore them if no HW BP is enabled. And if we want to be more clever, we can save DR{i} (i = 0, 1, 2, 3) only if HW BP i is enabled. I think this could be an important optimization, because this function is run before every VM entry, so even a few extra instructions could end up having a non-negligible performance penalty.

Most of the time, the vCPU is not in debugging mode [...], so we don't need to save/load debug registers

I'm curious about this: What happens after a VM-exit event if the guest has modified dr{0,1,2,3,6}? If "DR exiting" is disabled, this would have side-effects on the host, right?

So saving/loading drN registers should occur if any of these events occur:

"DR exiting" is disabled.

"HAX_DEBUG_USE_HW_BP" is enabled.

What do you think?

That's indeed a very good point. Should we always enable "DR exiting" though, because:

If the debugger runs on the host, we can prevent the guest from tampering with DRs.

If the debugger runs on the guest, we can keep track of the "dirty" DRs we should save/restore. This is more efficient than having "DR exiting" disabled and saving/restore all the DRs at every guest/host context switch.

If you agree with this approach, could you implement it in this PR (maybe as a separate commit)?

raphaelning · 2018-07-24T10:25:55Z

core/vcpu.c

@@ -1346,10 +1362,39 @@ void vcpu_load_host_state(struct vcpu_t *vcpu)

    load_host_msr(vcpu);
    set_cr2(hstate->hcr2);
+    set_dr0(hstate->dr0);


Make these operations conditional, so as to minimize the performance penalty.

raphaelning · 2018-07-24T10:27:44Z

core/vcpu.c

+    struct vcpu_state_t *state = vcpu->state;
+
+    save_guest_msr(vcpu);
+    state->_dr0 = get_dr0();


Make these operations conditional, so as to minimize the performance penalty.

raphaelning · 2018-07-24T10:28:53Z

core/vcpu.c

+    struct vcpu_state_t *state = vcpu->state;
+
+    load_guest_msr(vcpu);
+    set_dr0(state->_dr0);


Make these operations conditional, so as to minimize the performance penalty.

raphaelning · 2018-07-24T10:30:49Z

core/vcpu.c

@@ -2195,6 +2240,22 @@ static int exit_exc_nmi(struct vcpu_t *vcpu, struct hax_tunnel *htun)
            dump_vmcs(vcpu);
            break;
        }
+        case VECTOR_DB: {
+            hax_warning("#DB!\n");


Warning and error messages are visible to the end user, so let's use hax_info here instead.

I'll remove this line. It helped me on early stages, but has no use anymore.

raphaelning · 2018-07-24T15:52:10Z

core/vcpu.c

+        vcpu->debug_control = 0;
+        vmwrite(vcpu, GUEST_DR7, 0);
+        vmwrite(vcpu, GUEST_RFLAGS,
+            vmread(vcpu, GUEST_RFLAGS) & ~(1 << 8) /* TF */);


raphaelning · 2018-07-24T15:56:55Z

include/hax_interface.h

+            uint64_t rip;
+            uint64_t dr6;
+            uint64_t dr7;
+        } debug;


nit: I wonder if we should add a reserved field. Are there any debugging features we could support in the future that might require passing additional data to QEMU? Problem is we have reached the size of the union (24 bytes), but we could also use a second tunnel (vcpu->io_buf) like the fast MMIO interface does.

Are there any debugging features we could support in the future that might require passing additional data to QEMU?

I don't think so. On the SDM I've seen other nice features that could benefit debuggers (e.g. tracing), but they can have their own ioctl since they are too wildly different from breakpoints/stepping and considerably more complex.

So I don't think we will ever need more than these 24 bytes.

raphaelning · 2018-07-24T16:08:43Z

include/hax_interface.h

+struct hax_debug_t {
+    uint32_t control;
+    uint64_t dr[8];
+} PACKED;


PACKED is ignored for Windows builds, so I suspect there's an implicit uint32_t inserted after control. How about we manually declare it as uint32_t reserved? It'll also give us room for extension.

How about we manually declare it as uint32_t reserved?

Agreed.

raphaelning · 2018-07-24T16:11:26Z

windows/hax_entry.h

@@ -163,4 +163,8 @@ extern PDRIVER_OBJECT HaxDriverObject;
 #define HAX_VM_IOCTL_NOTIFY_QEMU_VERSION \
        CTL_CODE(HAX_DEVICE_TYPE, 0x910, METHOD_BUFFERED, FILE_ANY_ACCESS)

+/* API version 5.0 */
+#define HAX_IOCTL_VCPU_DEBUG \
+        CTL_CODE(HAX_DEVICE_TYPE, 0x912, METHOD_BUFFERED, FILE_ANY_ACCESS)


nit: I skipped 0x912 intentionally when I added the ADD_RAMBLOCK ioctl, because 0x910 appears twice... So I'd use 0x916 for this one, but that's not a big deal.

raphaelning · 2018-07-24T16:21:02Z

windows/hax_entry.h

@@ -163,4 +163,8 @@ extern PDRIVER_OBJECT HaxDriverObject;
 #define HAX_VM_IOCTL_NOTIFY_QEMU_VERSION \
        CTL_CODE(HAX_DEVICE_TYPE, 0x910, METHOD_BUFFERED, FILE_ANY_ACCESS)

+/* API version 5.0 */


The HAXM API is indeed versioned, and the current version is 4 (HAX_CUR_VERSION). When we update the API, we could bump the API version, but we could also add a "capability flag" (see API.md for details). I'd actually prefer the latter approach, since we're only adding one ioctl. As an example, please take a look at commit 43ff9f0.

raphaelning · 2018-07-24T16:25:14Z

In summary: guests can only be debugged from either host or guest, but not both simultaneously. I suppose that's not a big limitation

Not at all!

Sorry I'm yet to try this PR. It looks like I'll have to delay that for a couple more days... I've done a rough review of the code, and here are my comments so far, but I may revisit a few places later on more carefully :-)

AlexAltea · 2018-07-25T12:23:41Z

I've updated the patch. :-)

EDIT: As always, I've added them in an extra commit for easier reviewing, but I will squash everything into a single self-contained commit before merging.

raphaelning · 2018-07-30T10:50:18Z

Sorry for the delay in the review. I actually need your help with testing this PR... I had no problem building HAXM or QEMU with your changes, but I can't even get breakpoints working with KVM or TCG, so I haven't seen the "correct" behavior yet :-( Well, a few years back I used to debug the Android Emulator kernel on Ubuntu using QEMU GDB, but I haven't been lucky today.

Basically I'm trying to debug the Linux kernel, and I can set breakpoints using a symbol name (e.g. start_kernel), but they are never triggered. I'v tried both i386 and x86_64 kernels, both HW and SW breakpoints, both KVM and TCG, no luck yet. I've even followed this tutorial but the result didn't change. I don't know if that has anything to do with the fact that I'm using the latest QEMU and a recent Linux kernel (4.15/4.16).

Is it possible for me to reproduce your setup for testing this PR?

AlexAltea · 2018-08-04T15:54:49Z

Hi @raphaelning, I will be traveling until August 22th (due DEFCON + other matters), so it's difficult to setup a proper environment to work on HAXM at the moment.

Last week I encountered a weird bug on my guest kernel where hardware/software breakpoints were ignored. I fixed it by forcing by forcing #DB/#BP exits in the vcpu_update_exception_bitmap function (see patch below). I had to leave so I wasn't able to debug the issue, let alone pushing a proper fix yet.

-   exc_bitmap = (1u << VECTOR_MC) | (1u << VECTOR_NM);
+   exc_bitmap = (1u << VECTOR_MC) | (1u << VECTOR_NM) | (1u << VECTOR_DB) | (1u << VECTOR_BP);
+#if 0
    if (vcpu->debug_control & (HAX_DEBUG_USE_HW_BP | HAX_DEBUG_STEP)) {
        exc_bitmap |= (1u << VECTOR_DB);
    }
    if (vcpu->debug_control & HAX_DEBUG_USE_SW_BP) {
        exc_bitmap |= (1u << VECTOR_BP);
    }
+#endif

Try this patch if you can, and let me know if it solves your issues with Ubuntu. If so, I could try VNC'ing into my main machine and try to fix the problem.

PS: Also, note that if you apply software breakpoints before the kernel is even mapped, the int 3 patch will be overwritten by the kernel contents, which removes your software breakpoints. I've edited my ~/.gdbinit such that it sets a hardware breakpoint on the kernel entry point, and then it applies a predefined list of software breakpoints.

raphaelning · 2018-08-08T09:05:49Z

Thanks, and no problem at all--I also took some time off and just got back.

I did consider the fact that I might not be able to set SW breakpoints before the guest Linux kernel decompresses itself, and that GDB might not be able to detect the correct target architecture before the kernel jumps into 32- (for i386) or 64-bit (for x86_64) mode. So I've already tried setting HW breakpoints, and SW breakpoints well after seeing dmesg output, but the breakpoints still don't get hit... Again, I was testing against KVM/TCG, because I wanted to set up a "control group" first ;-) I'll debug that tomorrow.

There's also a very small chance that guest debugging support is broken for both KVM and TCG, so I'll try HAXM+GDB with your patch and hack too. I still wonder if I could use the same guest kernel, QEMU and GDB commands that you use; if not, I'll just stick with the latest Linux kernel and QEMU.

raphaelning · 2018-08-14T08:09:10Z

I was testing against KVM/TCG, because I wanted to set up a "control group" first

OK I just got this working. The key is to use an older Linux kernel as the guest, e.g. v3.10:

https://github.com/nathanchance/linux-stable/tree/linux-3.10.y

For some reason, QEMU GDB just can't break into recent x86 Linux kernels, regardless of the accelerator being used.

For the record, here's how to reproduce my setup:

Build Linux 3.10 using x86_64_defconfig + CONFIG_DEBUG_INFO.
Run qemu-system-x86_64 with at least the following options:
- -kernel /path/to/linux-stable/arch/x86/boot/bzImage
- -s -S
- -accel followed by the desired accelerator name
$ gdb /path/to/linux-stable/vmlinux

(gdb) target remote :1234

Remote debugging using :1234
0x000000000000fff0 in init_tss ()

(gdb) hb start_kernel

Hardware assisted breakpoint 1 at 0xffffffff81cd69d2: file init/main.c, line 473.

(gdb) c

Continuing.

Breakpoint 1, start_kernel () at init/main.c:473
473     {

Up to this point, HAXM works just as well as KVM and TCG. I've also verified that single-stepping (s/n) and SW breakpoints (b free_initmem) work as expected. I haven't seen the issue you described and haven't applied the unconditional #DB/#BP exiting hack.

However, I've noticed another bug: whenever I interrupt the guest manually using Ctrl-C, GDB always shows the last HW or SW breakpoint that was hit, instead of the current guest RIP:

(gdb) target remote :1234
Remote debugging using :1234
0x000000000000fff0 in init_tss ()
(gdb) hb start_kernel
Hardware assisted breakpoint 1 at 0xffffffff81cd69d2: file init/main.c, line 473.
(gdb) c
Continuing.

Breakpoint 1, start_kernel () at init/main.c:473
473     {
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0xffffffff81cd69d3 in start_kernel () at init/main.c:473
473     {
(gdb) b free_initmem
Breakpoint 2 at 0xffffffff8102fcc0: file arch/x86/mm/init.c, line 561.
(gdb) c
Continuing.

Breakpoint 2, free_initmem () at arch/x86/mm/init.c:561
561     {
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
free_initmem () at arch/x86/mm/init.c:562
562             free_init_pages("unused kernel memory",

junxiaoc · 2018-09-07T10:54:20Z

For this issue: "However, I've noticed another bug: whenever I interrupt the guest manually using Ctrl-C, GDB always shows the last HW or SW breakpoint that was hit, instead of the current guest RIP:"

It should be dirty flag setting issue in Qemu, and be a separate issue. I will do more investigation for a formal fix.

junxiaoc · 2018-09-13T07:17:06Z

Above Ctrl C issue should be register sync issue. It could be fixed by qemu change.

AlexAltea · 2018-09-15T13:50:44Z

@raphaelning @junxiaoc Thanks a lot for the feedback. And sorry for my late reply, but I have been busy for the past several weeks with other matters.

I've checked once again my patch and breakpoints/stepping works as expected (no further patches needed). As for the Ctrl+C issue, it seems @junxiaoc's patch for QEMU fixes it, so it's unrelated to this PR. I'll rebase the patch so that there's no conflicts.

raphaelning · 2018-09-17T02:52:48Z

No problem :-) We were unable to reproduce the breakpoints being ignored issue, so we also thought this PR is good to go.

Please go ahead with the rebase and git history cleanup, and then we'll merge this PR.

Signed-off-by: Alexandro Sanchez Bach <alexandro@phi.nz>

AlexAltea · 2018-09-17T03:08:41Z

All done!

The new DEBUG ioctl is identified by a feature flag, so it's not tied to a specific HAXM API version.

raphaelning · 2018-09-17T03:41:14Z

Thanks! I caught a couple of misleading comments, and then realized I could edit them out myself. I hope you're fine with that.

We'll run regular tests before doing a squash-merge.

mborgerson · 2018-09-24T21:57:55Z

Hi @raphaelning -- Are the changes @AlexAltea made to QEMU (AlexAltea/orbital-qemu@77d61c7) to enable guest debug planned to be upstreamed to mainline?

raphaelning · 2018-09-25T09:17:59Z

@AlexAltea is the original author of the QEMU-side patch, but he's okay with us upstreaming it. Our plan is to improve the HAXM-side patch and include it in a new HAXM release (7.4.0) before upstreaming the QEMU-side patch. @junxiaoc is working on a patch to conditionally save/restore DRs and monitor guest writes to DRs (left as TODOs by this PR). He and I have also rebased the QEMU-side patch onto QEMU 3.0.0 and fixed a few issues. If you'd like to try them, @junxiaoc can attach them here.

This change is to improve robustness and performance for debug feature(#81). DR registers are loaded to guest only if they are updated or current thread is migrated to other CPU core. Host DRs will not be saved and restored unless host process (QEMU) is debugged with hardware breakpoint. When a VCPU is being debugged by the host using HW breakpoint, prevent its DR registers from being tampered with by the guest OS or by SET_REGS ioctl. Verified: - Guest gdb only; - QEMU gdb (debugging guest from host) only; - Starting guest gdb, then using QEMU gdb to debug; - Host debugging QEMU + Guest gdb; - Host debugging QEMU + QEMU gdb; - Host debugging QEMU + QEMU gdb + Guest gdb QEMU gdb works, and guest gdb reports app crash. Please refer to known issues Known issue: - Guest gdb might report app crash When QEMU gdb is enabled This is probabaly because existed guest DR state is discarded as soon as QEMU gdb enables hardware breakpoint. Change-Id: I872ba70b2bc10a11a37425e4e0aa1adad20d093d Signed-off-by: Junxiao Chang <junxiao.chang@intel.com>

This change is to improve robustness and performance for debug feature(#81). DR registers are loaded to guest only if they are updated or current thread is migrated to other CPU core. Host DRs will not be saved and restored unless host process (QEMU) is debugged with hardware breakpoint. When a VCPU is being debugged by the host using HW breakpoint, prevent its DR registers from being tampered with by the guest OS or by SET_REGS ioctl. Verified: - Guest gdb only; - QEMU gdb (debugging guest from host) only; - Starting guest gdb, then using QEMU gdb to debug; - Host debugging QEMU + Guest gdb; - Host debugging QEMU + QEMU gdb; - Host debugging QEMU + QEMU gdb + Guest gdb QEMU gdb works, and guest gdb reports app crash. Please refer to known issues Known issue: - Guest gdb might report app crash When QEMU gdb is enabled This is probabaly because existed guest DR state is discarded as soon as QEMU gdb enables hardware breakpoint. Signed-off-by: Junxiao Chang <junxiao.chang@intel.com>

junxiaoc · 2018-11-01T04:15:06Z

patches_debug_feature_in_qemu.zip

Attached are qemu patches which includes "Ctrl C" fix, Mac fix, and patch which rebases to qemu 3.0.0.

AlexAltea force-pushed the debug branch 3 times, most recently from 07c0d78 to d938351 Compare July 22, 2018 22:58

raphaelning reviewed Jul 24, 2018

View reviewed changes

AlexAltea force-pushed the debug branch 2 times, most recently from 49c1a88 to c67b087 Compare September 17, 2018 03:07

Add guest debugging support

6a92ea5

Signed-off-by: Alexandro Sanchez Bach <alexandro@phi.nz>

AlexAltea force-pushed the debug branch from c67b087 to 6a92ea5 Compare September 17, 2018 03:07

raphaelning added 2 commits September 17, 2018 11:30

Undo unnecessary API version bump

e37664c

Remove API version comment

8bbc47d

The new DEBUG ioctl is identified by a feature flag, so it's not tied to a specific HAXM API version.

raphaelning merged commit 99bd00f into intel:master Sep 17, 2018

AlexAltea deleted the debug branch September 17, 2018 11:19

junxiaoc mentioned this pull request Oct 24, 2018

haxm debugging feature improvement #114

Merged

raphaelning mentioned this pull request Oct 31, 2018

Added support for Linux hosts #108

Merged

4 tasks

raphaelning mentioned this pull request Dec 10, 2018

No support for NetBSD/amd64 guest #136

Closed

AlexAltea mentioned this pull request Feb 23, 2019

Guest debugging support #66

Closed

mdepa mentioned this pull request Dec 9, 2019

Unable to boot ubuntu cloud-image with HAX enabled. #249

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add guest debugging support #81

Add guest debugging support #81

AlexAltea commented Jul 22, 2018 •

edited

Loading

raphaelning commented Jul 23, 2018

AlexAltea commented Jul 23, 2018 •

edited

Loading

raphaelning Jul 24, 2018

AlexAltea Jul 25, 2018 •

edited

Loading

raphaelning Jul 30, 2018

raphaelning Jul 24, 2018

raphaelning Jul 24, 2018

raphaelning Jul 24, 2018

raphaelning Jul 24, 2018

AlexAltea Jul 25, 2018

raphaelning Jul 24, 2018

raphaelning Jul 24, 2018

AlexAltea Jul 25, 2018

raphaelning Jul 24, 2018

AlexAltea Jul 25, 2018

raphaelning Jul 24, 2018

raphaelning Jul 24, 2018

raphaelning commented Jul 24, 2018

AlexAltea commented Jul 25, 2018 •

edited

Loading

raphaelning commented Jul 30, 2018

AlexAltea commented Aug 4, 2018 •

edited

Loading

raphaelning commented Aug 8, 2018

raphaelning commented Aug 14, 2018 •

edited

Loading

junxiaoc commented Sep 7, 2018

junxiaoc commented Sep 13, 2018 •

edited

Loading

AlexAltea commented Sep 15, 2018 •

edited

Loading

raphaelning commented Sep 17, 2018

AlexAltea commented Sep 17, 2018

raphaelning commented Sep 17, 2018

mborgerson commented Sep 24, 2018

raphaelning commented Sep 25, 2018

junxiaoc commented Nov 1, 2018

Add guest debugging support #81

Add guest debugging support #81

Conversation

AlexAltea commented Jul 22, 2018 • edited Loading

Motivation

Design

Interface

Features

Documentation

Samples

raphaelning commented Jul 23, 2018

AlexAltea commented Jul 23, 2018 • edited Loading

Choose a reason for hiding this comment

AlexAltea Jul 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raphaelning commented Jul 24, 2018

AlexAltea commented Jul 25, 2018 • edited Loading

raphaelning commented Jul 30, 2018

AlexAltea commented Aug 4, 2018 • edited Loading

raphaelning commented Aug 8, 2018

raphaelning commented Aug 14, 2018 • edited Loading

junxiaoc commented Sep 7, 2018

junxiaoc commented Sep 13, 2018 • edited Loading

AlexAltea commented Sep 15, 2018 • edited Loading

raphaelning commented Sep 17, 2018

AlexAltea commented Sep 17, 2018

raphaelning commented Sep 17, 2018

mborgerson commented Sep 24, 2018

raphaelning commented Sep 25, 2018

junxiaoc commented Nov 1, 2018

AlexAltea commented Jul 22, 2018 •

edited

Loading

AlexAltea commented Jul 23, 2018 •

edited

Loading

AlexAltea Jul 25, 2018 •

edited

Loading

AlexAltea commented Jul 25, 2018 •

edited

Loading

AlexAltea commented Aug 4, 2018 •

edited

Loading

raphaelning commented Aug 14, 2018 •

edited

Loading

junxiaoc commented Sep 13, 2018 •

edited

Loading

AlexAltea commented Sep 15, 2018 •

edited

Loading