This phase I/II trial studies the side effects and best dose of adding hydroxychloroquine to dabrafenib and/or trametinib, and to see how well they work in treating children with low grade or high grade brain tumors previously treated with similar drugs that did not respond completely (progressive) or tumors that came back while receiving a similar agent (recurrent). Patients must also have specific genetic mutations including BRAF V600 mutations or BRAF fusion/duplication, with or without neurofibromatosis type 1. Neurofibromatosis type 1 is an inherited genetic condition that causes tumors to grow on nerve tissue. Drugs used in chemotherapy, such as hydroxychloroquine, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Trametinib and dabrafenib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Giving hydroxychloroquine with trametinib and/or dabrafenib may lower the chance of brain tumors from growing or spreading compared to usual treatments.